-
Notifications
You must be signed in to change notification settings - Fork 178
Add udp2, an experimental IPC layer implementation #1357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
contrib/udp2/ic_common/README.md
Outdated
|
||
``` | ||
cd existing_repo | ||
git remote add origin https://code.hashdata.xyz/cloudberry/plugins/interconnect.git |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use github link
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fix in 251adf4
/*------------------------------------------------------------------------- | ||
* ic_arrow_adapter.cpp | ||
* C++ adapter interface for Arrow. | ||
* | ||
* Portions Copyright (c) 2023, HashData Technology Limited. | ||
* | ||
* | ||
* IDENTIFICATION | ||
* contrib/interconnect/udp/ic_arrow_adapter.cpp | ||
* | ||
*------------------------------------------------------------------------- | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to use the standard ASF license header, e.g:
cloudberry/contrib/pgcrypto/sm3.h
Lines 1 to 26 in f61fbd9
/*------------------------------------------------------------------------- | |
* | |
* Licensed to the Apache Software Foundation (ASF) under one | |
* or more contributor license agreements. See the NOTICE file | |
* distributed with this work for additional information | |
* regarding copyright ownership. The ASF licenses this file | |
* to you under the Apache License, Version 2.0 (the | |
* "License"); you may not use this file except in compliance | |
* with the License. You may obtain a copy of the License at | |
* | |
* http://www.apache.org/licenses/LICENSE-2.0 | |
* | |
* Unless required by applicable law or agreed to in writing, | |
* software distributed under the License is distributed on an | |
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | |
* KIND, either express or implied. See the License for the | |
* specific language governing permissions and limitations | |
* under the License. | |
* | |
* sm3.h | |
* | |
* IDENTIFICATION | |
* contrib/pgcrypto/sm3.h | |
* | |
*------------------------------------------------------------------------- | |
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be added for the original created files. For the third-party files, we need to keep the original license headers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fix in 251adf4
Awesome! Could you please share some information about the architecture of the new solution? I think there might be an original (acmsigmod?) article on how to build an effective interconnect layer. (I see some familiar elements in the code, but I cannot fully comprehend what it is) |
During performance testing, we identified the Motion node as a significant performance bottleneck. Further analysis revealed that the root cause stems from inefficiencies in the underlying IPC layer protocol implementation. To address this, we plan to enhance the IPC layer to improve overall system performance. Currently, the IPC layer is tightly coupled with the database kernel, which makes implementing performance optimizations challenging and limits flexibility. As an initial step, we are decoupling the IPC layer from the database kernel. This separation will not only simplify future improvements but also enable independent testing and development of the IPC layer. This commit represents foundational work—specifically, the decoupling of the existing UDP-based IPC layer from the database kernel—and lays the groundwork for subsequent performance optimization efforts. |
The following two files also need the license header:
|
Fixes #ISSUE_Number
What does this PR do?
Type of Change
Breaking Changes
Test Plan
make installcheck
make -C src/test installcheck-cbdb-parallel
Impact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions