Skip to content

ingest: fix default region split size and keys in nextgen#64540

Merged
ti-chi-bot[bot] merged 3 commits into
pingcap:masterfrom
D3Hunter:fix-hit-split-limit-on-pd-ha
Nov 19, 2025
Merged

ingest: fix default region split size and keys in nextgen#64540
ti-chi-bot[bot] merged 3 commits into
pingcap:masterfrom
D3Hunter:fix-hit-split-limit-on-pd-ha

Conversation

@D3Hunter

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #64526, ref #61702

Problem Summary:

What changed and how does it work?

nextgen has different setting for region split.

since nextgen we only support global sort, and each ingest subtask is about 100G, so max 100 regions on default setting, will not exceed the limit mentioned in the issue

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
apply below diff, so we always fail to get the region split setting from PD

diff --git a/pkg/executor/importer/table_import.go b/pkg/executor/importer/table_import.go
index a8e0d737ee..0affdf0d73 100644
--- a/pkg/executor/importer/table_import.go
+++ b/pkg/executor/importer/table_import.go
@@ -161,7 +161,7 @@ func GetRegionSplitSizeKeys(ctx context.Context) (regionSplitSize int64, regionS
 		return 0, 0, errors.Trace(err)
 	}
 	defer pdCli.Close()
-	return local.GetRegionSplitSizeKeys(ctx, pdCli, tls)
+	return 0, 0, errors.New("not implemented")
 }
 
 // NewTableImporter creates a new table importer.
diff --git a/pkg/lightning/backend/local/local.go b/pkg/lightning/backend/local/local.go
index f5b7037d9d..ffa5c7d9b2 100644
--- a/pkg/lightning/backend/local/local.go
+++ b/pkg/lightning/backend/local/local.go
@@ -1865,9 +1865,9 @@ func GetRegionSplitSizeKeys(ctx context.Context, cli pd.Client, tls *common.TLS)
 			StatusAddr: store.StatusAddress,
 		}
 		serverInfo.ResolveLoopBackAddr()
-		regionSplitSize, regionSplitKeys, err := getSplitConfFromStoreFunc(ctx, serverInfo.StatusAddr, tls)
+		_, _, err := getSplitConfFromStoreFunc(ctx, serverInfo.StatusAddr, tls)
 		if err == nil {
-			return regionSplitSize, regionSplitKeys, nil
+			return 0, 0, errors.New("get region split size and keys failed")
 		}
 		tidblogutil.Logger(ctx).Warn("get region split size and keys failed", zap.Error(err), zap.String("store", serverInfo.StatusAddr))
 	}

in nextgen

[2025/11/18 20:05:03.939 +08:00] [INFO] [planner.go:617] ["split kv range with split size and keys"] [keyspaceName=SYSTEM] [task-id=1] [task-key=SYSTEM/ImportInto/1] [region-split-size=1073741824] [region-split-keys=102400000] [range-size=214748365] [range-keys=20480000]
[2025/11/18 20:05:08.374 +08:00] [INFO] [backfilling_dist_scheduler.go:766] ["split kv range with split size and keys"] [keyspaceName=SYSTEM] [category=ddl-ingest] [region-split-size=1073741824] [region-split-keys=102400000] [range-size=214748365] [range-keys=20480000]

in classic

[2025/11/18 20:10:18.726 +08:00] [INFO] [planner.go:617] ["split kv range with split size and keys"] [task-id=1] [task-key=ImportInto/1] [region-split-size=100663296] [region-split-keys=960000] [range-size=201326592] [range-keys=1920000]
[2025/11/18 20:10:23.478 +08:00] [INFO] [backfilling_dist_scheduler.go:766] ["split kv range with split size and keys"] [category=ddl-ingest] [region-split-size=100663296] [region-split-keys=960000] [range-size=201326592] [range-keys=1920000]
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 18, 2025
@tiprow

tiprow Bot commented Nov 18, 2025

Copy link
Copy Markdown

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@D3Hunter

Copy link
Copy Markdown
Contributor Author

/cherry-pick release-nextgen-20251011

@ti-chi-bot

Copy link
Copy Markdown
Member

@D3Hunter: once the present PR merges, I will cherry-pick it on top of release-nextgen-20251011 in the new PR and assign it to you.

Details

In response to this:

/cherry-pick release-nextgen-20251011

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@codecov

codecov Bot commented Nov 18, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 75.0338%. Comparing base (88177db) to head (b9f0fed).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #64540        +/-   ##
================================================
+ Coverage   72.6887%   75.0338%   +2.3450%     
================================================
  Files          1866       1889        +23     
  Lines        506414     516318      +9904     
================================================
+ Hits         368106     387413     +19307     
+ Misses       115929     105172     -10757     
- Partials      22379      23733      +1354     
Flag Coverage Δ
integration 48.2252% <0.0000%> (?)
unit 72.6373% <92.8571%> (+0.3753%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.8700% <ø> (ø)
parser ∅ <ø> (∅)
br 63.6055% <ø> (+17.4139%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@D3Hunter D3Hunter changed the title ingest: fix default region split setting in nextgen ingest: fix default region split size and keys in nextgen Nov 18, 2025
@D3Hunter

Copy link
Copy Markdown
Contributor Author

/retest

@tiprow

tiprow Bot commented Nov 18, 2025

Copy link
Copy Markdown

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 19, 2025
@ti-chi-bot

ti-chi-bot Bot commented Nov 19, 2025

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fzzf678, wjhuang2016

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 19, 2025
@ti-chi-bot

ti-chi-bot Bot commented Nov 19, 2025

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2025-11-19 02:49:41.365859881 +0000 UTC m=+66345.015054338: ☑️ agreed by fzzf678.
  • 2025-11-19 04:39:04.506691646 +0000 UTC m=+72908.155886103: ☑️ agreed by wjhuang2016.

@ti-chi-bot ti-chi-bot Bot merged commit 17945b9 into pingcap:master Nov 19, 2025
32 checks passed
@ti-chi-bot

Copy link
Copy Markdown
Member

@D3Hunter: new pull request created to branch release-nextgen-20251011: #64554.

Details

In response to this:

/cherry-pick release-nextgen-20251011

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tidb-x] import into failed with error "split region failed" when simulated pd leader network partition

4 participants