Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Write combined txn log parallel (backport #55143) #55445

Merged
merged 2 commits into from
Jan 26, 2025

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Jan 26, 2025

Why I'm doing:

If the import task involves multiple partitions, the combined transaction log will be written one by one, which may slowdown the import speed.

What I'm doing:

Write combined txn log parallel

write combined txn log for 100 partitions.
before this pr

I20250110 15:44:43.331159 139757477246720 tablet_sink_sender.cpp:384] write_combined_txn_log non-parallel cost time: 1545

after this pr

I20250110 15:46:47.941779 139757477246720 tablet_sink_sender.cpp:376] write_combined_txn_log parallel cost time: 140

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Signed-off-by: sevev <qiangzh95@gmail.com>
(cherry picked from commit af3d78f)

# Conflicts:
#	be/src/storage/lake/tablet_manager.cpp
Copy link
Contributor Author

mergify bot commented Jan 26, 2025

Cherry-pick of af3d78f has failed:

On branch mergify/bp/branch-3.3/pr-55143
Your branch is up to date with 'origin/branch-3.3'.

You are currently cherry-picking commit af3d78f12f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/common/config.h
	modified:   be/src/exec/tablet_sink_sender.cpp
	modified:   be/src/exec/write_combined_txn_log.cpp
	modified:   be/src/exec/write_combined_txn_log.h
	modified:   be/src/runtime/exec_env.cpp
	modified:   be/src/runtime/exec_env.h
	modified:   be/test/CMakeLists.txt
	new file:   be/test/storage/lake/write_combined_txn_log_test.cpp

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/src/storage/lake/tablet_manager.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify mergify bot added the conflicts label Jan 26, 2025
@wanpengfei-git wanpengfei-git enabled auto-merge (squash) January 26, 2025 03:12
@mergify mergify bot closed this Jan 26, 2025
auto-merge was automatically disabled January 26, 2025 03:13

Pull request was closed

Copy link
Contributor Author

mergify bot commented Jan 26, 2025

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

@sevev sevev reopened this Jan 26, 2025
@wanpengfei-git wanpengfei-git enabled auto-merge (squash) January 26, 2025 03:18
Signed-off-by: sevev <qiangzh95@gmail.com>
@wanpengfei-git wanpengfei-git merged commit b411fa4 into branch-3.3 Jan 26, 2025
28 of 29 checks passed
@wanpengfei-git wanpengfei-git deleted the mergify/bp/branch-3.3/pr-55143 branch January 26, 2025 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants