[BugFix] Support force cancel refresh materialized view & optimize some task run strategies #46131

LiShuMing · 2024-05-23T08:05:17Z

Why I'm doing:

Little bugs in mv refreshing.

What I'm doing:

Support force cancel refresh materialized view which can ensure the running task run can be removed from running queue to avoid occupying resources too long.
Add Config. enable_mv_refresh_sync_refresh_mergeable to support merge sync refresh mv task;
Add Config. max_mv_task_run_meta_message_values_length to limit the mv task run extra message's size;
Make ENABLE_INSERT_STRICT and ENABLE_PROFILE configurage in mv refresh.
Change filterRows to long since be's type is int64_t and it may be overflow .

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

fe/fe-core/src/main/java/com/starrocks/scheduler/persist/MVTaskRunExtraMessage.java

…rategy Signed-off-by: shuming.li <ming.moriarty@gmail.com>

sonarqubecloud · 2024-06-27T07:59:50Z

Quality Gate passed

Issues
23 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

packy92 · 2024-06-27T08:30:41Z

fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunManager.java

+        } finally {
+            // if it's force, remove it from running TaskRun map no matter it's killed or not
+            if (force) {
+                taskRunScheduler.removeRunningTask(taskRun.getTaskId());


why do this can help to avoid occupying resources too long?

The running queue is limited by Conf.task_runs_concurrency which is 4 by default. If one task run is always in the running queue, the task run scheduler can not schedule other tasks for the limit.

If the task run is in the running queue, other dependent tasks (triggerred by this task) will also not be scheduled.

nshangyiming · 2024-06-27T09:12:21Z

fe/fe-core/src/main/java/com/starrocks/common/Config.java

@@ -2830,6 +2830,13 @@ public class Config extends ConfigBase {
    @ConfField(mutable = true, comment = "The default try lock timeout for mv refresh to try base table/mv dbs' lock")
    public static int mv_refresh_try_lock_timeout_ms = 30 * 1000;

+    @ConfField(mutable = true, comment = "Whether enable to refresh materialized view in sync mode mergeable or not")
+    public static boolean enable_mv_refresh_sync_refresh_mergeable = false;


same prefix is better

github-actions · 2024-06-27T09:36:15Z

[FE Incremental Coverage Report]

✅ pass : 64 / 73 (87.67%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/qe/DDLStmtExecutor.java	0	1	00.00%	[388]
🔵	com/starrocks/scheduler/TaskManager.java	7	11	63.64%	[191, 279, 283, 284]
🔵	com/starrocks/sql/ast/CancelRefreshMaterializedViewStmt.java	2	3	66.67%	[25]
🔵	com/starrocks/scheduler/TaskRunFIFOQueue.java	8	10	80.00%	[145, 150]
🔵	com/starrocks/scheduler/persist/MVTaskRunExtraMessage.java	6	7	85.71%	[171]
🔵	com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java	7	7	100.00%	[]
🔵	com/starrocks/scheduler/persist/TaskRunStatus.java	2	2	100.00%	[]
🔵	com/starrocks/common/Config.java	2	2	100.00%	[]
🔵	com/starrocks/server/LocalMetastore.java	6	6	100.00%	[]
🔵	com/starrocks/scheduler/TaskRunManager.java	8	8	100.00%	[]
🔵	com/starrocks/scheduler/Constants.java	2	2	100.00%	[]
🔵	com/starrocks/connector/CatalogConnectorMetadata.java	1	1	100.00%	[]
🔵	com/starrocks/qe/StmtExecutor.java	3	3	100.00%	[]
🔵	com/starrocks/sql/optimizer/rule/transformation/materialization/MvUtils.java	6	6	100.00%	[]
🔵	com/starrocks/scheduler/ExecuteOption.java	3	3	100.00%	[]
🔵	com/starrocks/scheduler/TaskRunScheduler.java	1	1	100.00%	[]

github-actions · 2024-06-27T09:36:17Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-06-27T09:36:30Z

@Mergifyio backport branch-3.3

github-actions · 2024-06-27T09:36:32Z

@Mergifyio backport branch-3.2

github-actions · 2024-06-27T09:36:33Z

@Mergifyio backport branch-3.1

mergify · 2024-06-27T09:36:35Z

backport branch-3.3

✅ Backports have been created

#47586 [BugFix] Support force cancel refresh materialized view & optimize some task run strategies (backport #46131) has been created for branch branch-3.3 but encountered conflicts

mergify · 2024-06-27T09:36:36Z

backport branch-3.2

✅ Backports have been created

#47584 [BugFix] Support force cancel refresh materialized view & optimize some task run strategies (backport #46131) has been created for branch branch-3.2 but encountered conflicts

mergify · 2024-06-27T09:36:40Z

backport branch-3.1

✅ Backports have been created

#47585 [BugFix] Support force cancel refresh materialized view & optimize some task run strategies (backport #46131) has been created for branch branch-3.1 but encountered conflicts

…me task run strategies (#46131) Signed-off-by: shuming.li <ming.moriarty@gmail.com> (cherry picked from commit 02fb889) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/common/Config.java # fe/fe-core/src/main/java/com/starrocks/scheduler/ExecuteOption.java # fe/fe-core/src/main/java/com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java # fe/fe-core/src/main/java/com/starrocks/scheduler/persist/MVTaskRunExtraMessage.java # fe/fe-core/src/main/java/com/starrocks/scheduler/persist/TaskRunStatus.java # fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/rule/transformation/materialization/MvUtils.java

…me task run strategies (#46131) Signed-off-by: shuming.li <ming.moriarty@gmail.com> (cherry picked from commit 02fb889) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java

…me task run strategies (#46131) Signed-off-by: shuming.li <ming.moriarty@gmail.com> (cherry picked from commit 02fb889) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/connector/CatalogConnectorMetadata.java # fe/fe-core/src/main/java/com/starrocks/scheduler/ExecuteOption.java # fe/fe-core/src/main/java/com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java # fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunManager.java # fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunScheduler.java # fe/fe-core/src/main/java/com/starrocks/scheduler/persist/MVTaskRunExtraMessage.java # fe/fe-core/src/main/java/com/starrocks/scheduler/persist/TaskRunStatus.java # fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/rule/transformation/materialization/MvUtils.java # fe/fe-core/src/test/java/com/starrocks/connector/CatalogConnectorMetadataTest.java # fe/fe-core/src/test/java/com/starrocks/scheduler/TaskManagerTest.java

…me task run strategies (backport #46131) (#47584) Signed-off-by: shuming.li <ming.moriarty@gmail.com> Co-authored-by: shuming.li <ming.moriarty@gmail.com>

…me task run strategies (backport #46131) (#47586) Signed-off-by: shuming.li <ming.moriarty@gmail.com> Co-authored-by: shuming.li <ming.moriarty@gmail.com>

…me task run strategies (backport #46131) (#47585) Signed-off-by: shuming.li <ming.moriarty@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: shuming.li <ming.moriarty@gmail.com>

wanpengfei-git added the META-REVIEW label May 23, 2024

starrocks-cr bot reviewed May 23, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/scheduler/persist/MVTaskRunExtraMessage.java Show resolved Hide resolved

wanpengfei-git requested a review from a team May 23, 2024 08:05

mergify bot assigned LiShuMing May 23, 2024

github-actions bot added 3.3 3.2 3.1 labels May 23, 2024

LiShuMing changed the title ~~[BugFix] Fix task run state in multi fe replay~~ [BugFix] Avoid too much task run metadata in mv extra message May 23, 2024

LiShuMing force-pushed the fix/main/fix_mv_extra_message branch from a772ac9 to a46d210 Compare May 23, 2024 12:04

LiShuMing force-pushed the fix/main/fix_mv_extra_message branch from a46d210 to 42fd625 Compare June 19, 2024 10:28

LiShuMing requested review from a team as code owners June 19, 2024 10:28

LiShuMing changed the title ~~[BugFix] Avoid too much task run metadata in mv extra message~~ [UT] [BugFix] Avoid too much task run metadata in mv extra message Jun 26, 2024

LiShuMing force-pushed the fix/main/fix_mv_extra_message branch from 42fd625 to ffe0fc5 Compare June 26, 2024 09:36

LiShuMing requested review from a team as code owners June 26, 2024 09:36

LiShuMing force-pushed the fix/main/fix_mv_extra_message branch from ffe0fc5 to 67bb6ba Compare June 26, 2024 09:37

LiShuMing changed the title ~~[UT] [BugFix] Avoid too much task run metadata in mv extra message~~ [UT] [BugFix] Support force cancel refresh materialized view & optimize task run strategy Jun 26, 2024

LiShuMing force-pushed the fix/main/fix_mv_extra_message branch from 67bb6ba to 416ac30 Compare June 26, 2024 13:32

LiShuMing changed the title ~~[UT] [BugFix] Support force cancel refresh materialized view & optimize task run strategy~~ [BugFix] Support force cancel refresh materialized view & optimize task run strategy Jun 26, 2024

LiShuMing changed the title ~~[BugFix] Support force cancel refresh materialized view & optimize task run strategy~~ [BugFix] Support force cancel refresh materialized view & optimize some task run strategies Jun 26, 2024

Support force cancel refresh materialized view & optimize task run st…

d7da531

…rategy Signed-off-by: shuming.li <ming.moriarty@gmail.com>

LiShuMing force-pushed the fix/main/fix_mv_extra_message branch from f1e01fe to d7da531 Compare June 27, 2024 07:24

kangkaisen approved these changes Jun 27, 2024

View reviewed changes

packy92 reviewed Jun 27, 2024

View reviewed changes

satanson approved these changes Jun 27, 2024

View reviewed changes

LiShuMing enabled auto-merge (squash) June 27, 2024 08:56

Astralidea approved these changes Jun 27, 2024

View reviewed changes

nshangyiming approved these changes Jun 27, 2024

View reviewed changes

nshangyiming reviewed Jun 27, 2024

View reviewed changes

stephen-shelby approved these changes Jun 27, 2024

View reviewed changes

packy92 approved these changes Jun 27, 2024

View reviewed changes

LiShuMing merged commit 02fb889 into StarRocks:main Jun 27, 2024
66 of 68 checks passed

LiShuMing deleted the fix/main/fix_mv_extra_message branch June 27, 2024 09:36

github-actions bot removed the 3.3 label Jun 27, 2024

github-actions bot removed 3.2 3.1 labels Jun 27, 2024

github-actions bot added the 3.2-merged label Jun 27, 2024

github-actions bot added the 3.3-merged label Jun 27, 2024

github-actions bot added the 3.1-merged label Jun 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] Support force cancel refresh materialized view & optimize some task run strategies #46131

[BugFix] Support force cancel refresh materialized view & optimize some task run strategies #46131

LiShuMing commented May 23, 2024 •

edited

Loading

sonarqubecloud bot commented Jun 27, 2024

packy92 Jun 27, 2024

LiShuMing Jun 27, 2024

nshangyiming Jun 27, 2024

github-actions bot commented Jun 27, 2024

github-actions bot commented Jun 27, 2024

github-actions bot commented Jun 27, 2024

github-actions bot commented Jun 27, 2024

github-actions bot commented Jun 27, 2024

mergify bot commented Jun 27, 2024 •

edited

Loading

mergify bot commented Jun 27, 2024 •

edited

Loading

mergify bot commented Jun 27, 2024 •

edited

Loading

[BugFix] Support force cancel refresh materialized view & optimize some task run strategies #46131

[BugFix] Support force cancel refresh materialized view & optimize some task run strategies #46131

Conversation

LiShuMing commented May 23, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarqubecloud bot commented Jun 27, 2024

Quality Gate passed

packy92 Jun 27, 2024

Choose a reason for hiding this comment

LiShuMing Jun 27, 2024

Choose a reason for hiding this comment

nshangyiming Jun 27, 2024

Choose a reason for hiding this comment

github-actions bot commented Jun 27, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Jun 27, 2024

[BE Incremental Coverage Report]

github-actions bot commented Jun 27, 2024

github-actions bot commented Jun 27, 2024

github-actions bot commented Jun 27, 2024

mergify bot commented Jun 27, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Jun 27, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Jun 27, 2024 • edited Loading

✅ Backports have been created

LiShuMing commented May 23, 2024 •

edited

Loading

mergify bot commented Jun 27, 2024 •

edited

Loading

mergify bot commented Jun 27, 2024 •

edited

Loading

mergify bot commented Jun 27, 2024 •

edited

Loading