[refactor](jni) unified jni framework for jdbc catalog #26317
Conversation
|
clang-tidy review says "All clean, LGTM! 👍" |
zy-kkk
force-pushed
the
unified_jni_fra_for_jdbc_catalog
branch
from
321f2a8 to
195d61c
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
zy-kkk
force-pushed
the
unified_jni_fra_for_jdbc_catalog
branch
from
195d61c to
51069d7
Compare
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
zy-kkk
force-pushed
the
unified_jni_fra_for_jdbc_catalog
branch
from
51069d7 to
129f3a1
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
|
LGTM |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
|
Please finish the |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
zy-kkk
force-pushed
the
unified_jni_fra_for_jdbc_catalog
branch
from
129f3a1 to
9fd3ecd
Compare
|
run buildall |
github-actions
bot
removed
the
approved
|
TeamCity be ut coverage result: |
|
run buildall |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
zy-kkk
force-pushed
the
unified_jni_fra_for_jdbc_catalog
branch
from
646a9cb to
52a9f0c
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
run buildall |
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
run buildall |
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
zy-kkk
force-pushed
the
unified_jni_fra_for_jdbc_catalog
branch
from
52a9f0c to
25c4924
Compare
|
run buildall |
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
TeamCity be ut coverage result: |
This commit overhauls the JDBC connector logic within our project, transitioning from the previous mechanism of fetching data through JNI calls for individual ResultSet items to a more efficient and unified approach using the VectorTable data structure.
This commit overhauls the JDBC connector logic within our project, transitioning from the previous mechanism of fetching data through JNI calls for individual ResultSet items to a more efficient and unified approach using the VectorTable data structure.
Proposed changes
Issue Number: close #xxx
This commit overhauls the JDBC connector logic within our project, transitioning from the previous mechanism of fetching data through JNI calls for individual
ResultSetitems to a more efficient and unified approach using theVectorTabledata structure.Key Changes:
ResultSetinto aList<Object[]>. Then, our C++ code withinjdbc_connectorhad to read data from each column separately, making JNI calls to Java for necessary data format conversions and alignment for memory access.VectorTablememory address to directly fill theBlockstructure.Benefits:
VectorTablesignificantly declutters the data fetching and conversion logic on the Backend (BE) side for JDBC Catalog. C++ code is now cleaner and more maintainable, with a clear separation of responsibilities.VectorTableallows for faster data processing, better memory management, and overall improved read performance from JDBC sources.This refactor is part of our ongoing commitment to improve the architecture and performance of our system. By streamlining the data path from JDBC to our internal data structures, we expect to see more responsive data operations and lower latency in data handling tasks.
Impact:
The changes are expected to be backward compatible with existing JDBC data sources. However, thorough testing is recommended to ensure that all edge cases are handled correctly.
We encourage the community to test this new implementation and provide feedback on any issues or performance improvements observed.
Pending Tasks:
We aim to address this in future updates to further enhance the efficiency and performance of our system when dealing with these complex data types. By doing so, we anticipate reducing the overhead and potential bottlenecks associated with the current casting process, thus streamlining the entire data flow from JDBC to our internal representations.
Community involvement is crucial in this phase. We welcome contributions and suggestions on how to best approach this optimization for specialized data types. Your input will be invaluable in shaping the next iteration of our JDBC connector.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...