Implement push down instr function to tiflash #6081

lizhenhuan · 2022-10-06T15:10:21Z

What problem does this PR solve?

Issue Number: close #5109

Related PR:pingcap/tidb#35753

Problem Summary:Implement instr function push down to tiflash
Related PR:pingcap/tidb#38309

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Implement push down instr function to tiflash

Merge from pingcap repo

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

Merge from pingcap repo

Signed-off-by: lizhenhuan <1916038084@qq.com>

ti-chi-bot · 2022-10-06T15:10:23Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

ywqzzy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

lizhenhuan · 2022-10-06T15:15:36Z

/run-all-tests

sre-bot · 2022-10-06T15:33:27Z

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Flash/Coprocessor/DAGUtils.cpp                 347               124    64.27%          45                 6    86.67%         638               240    62.38%         412               158    61.65%
Functions/FunctionsString.cpp                 1616               396    75.50%         306                76    75.16%        3456               822    76.22%        1002               331    66.97%
Functions/tests/gtest_string_instr.cpp          29                15    48.28%           5                 1    80.00%          41                 3    92.68%           8                 8     0.00%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                         1992               535    73.14%         356                83    76.69%        4135              1065    74.24%        1422               497    65.05%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18404      7486             59.32%    215439  77119        64.20%

full coverage report (for internal network access only)

lizhenhuan · 2022-10-08T07:08:07Z

/run-all-tests

Signed-off-by: lizhenhuan <1916038084@qq.com>

sre-bot · 2022-10-08T07:21:25Z

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Flash/Coprocessor/DAGUtils.cpp                 347               124    64.27%          45                 6    86.67%         638               240    62.38%         412               158    61.65%
Functions/FunctionsString.cpp                 1616               396    75.50%         306                76    75.16%        3456               822    76.22%        1002               331    66.97%
Functions/tests/gtest_string_instr.cpp          29                15    48.28%           5                 1    80.00%          41                 3    92.68%           8                 8     0.00%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                         1992               535    73.14%         356                83    76.69%        4135              1065    74.24%        1422               497    65.05%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18404      7486             59.32%    215439  77065        64.23%

full coverage report (for internal network access only)

Signed-off-by: lizhenhuan <1916038084@qq.com>

dbms/src/Functions/tests/gtest_strings_instr.cpp

tests/fullstack-test/expr/instr.test

tests/fullstack-test/expr/instr_utf8.test

Signed-off-by: lizhenhuan <1916038084@qq.com>

lizhenhuan · 2022-10-23T13:48:42Z

/run-all-tests

Signed-off-by: lizhenhuan <1916038084@qq.com>

lizhenhuan · 2022-10-24T09:21:59Z

/run-all-tests

Signed-off-by: lizhenhuan <1916038084@qq.com>

lizhenhuan · 2022-10-24T10:11:09Z

/run-all-tests

lizhenhuan · 2022-10-24T10:16:51Z

/run-all-tests

lizhenhuan · 2022-10-24T10:37:01Z

/run-all-tests

lizhenhuan · 2022-10-24T10:37:31Z

/run-build

lizhenhuan · 2022-10-24T11:06:39Z

/run-build

lizhenhuan · 2022-10-25T00:31:36Z

/run-integration-test

lizhenhuan · 2022-10-25T00:32:27Z

/build

dbms/src/Functions/FunctionsString.cpp

Signed-off-by: lizhenhuan <1916038084@qq.com>

lizhenhuan · 2022-10-29T14:01:35Z

/run-all-tests

xzhangxian1008 · 2022-10-31T00:46:24Z

LGTM

lizhenhuan · 2022-10-31T03:05:24Z

/run-all-tests

Signed-off-by: lizhenhuan <1916038084@qq.com>

…instr_push_down

lizhenhuan · 2022-10-31T06:11:22Z

/run-all-tests

dbms/src/Functions/FunctionsString.cpp

Signed-off-by: lizhenhuan <1916038084@qq.com>

lizhenhuan · 2022-10-31T07:52:19Z

/run-all-tests

SeaRise · 2022-10-31T10:06:23Z

dbms/src/Functions/FunctionsString.cpp

+            const auto * substr_const_string = checkAndGetColumnConst<ColumnString>(c1_col_column.get());
+            if (!substr_const_string)
+            {
+                const auto * substr_vector_string = checkAndGetColumn<ColumnString>(c1_col_column.get());
+                const auto * str_const_string_value = checkAndGetColumn<ColumnString>(str_const_string->getDataColumnPtr().get());
+                constVector(str_const_string_value->getChars(), str_const_string_value->getOffsets(), substr_vector_string->getChars(), substr_vector_string->getOffsets(), val_num, res_vec);
+            }


Suggested change

const auto * substr_const_string = checkAndGetColumnConst<ColumnString>(c1_col_column.get());

if (!substr_const_string)

{

const auto * substr_vector_string = checkAndGetColumn<ColumnString>(c1_col_column.get());

const auto * str_const_string_value = checkAndGetColumn<ColumnString>(str_const_string->getDataColumnPtr().get());

constVector(str_const_string_value->getChars(), str_const_string_value->getOffsets(), substr_vector_string->getChars(), substr_vector_string->getOffsets(), val_num, res_vec);

}

if (const auto * substr_vector_string = checkAndGetColumn<ColumnString>(c1_col_column.get()))

{

const auto * str_const_string_value = checkAndGetColumn<ColumnString>(str_const_string->getDataColumnPtr().get());

constVector(str_const_string_value->getChars(), str_const_string_value->getOffsets(), substr_vector_string->getChars(), substr_vector_string->getOffsets(), val_num, res_vec);

}

else

throw Exception(

fmt::format("Illegal column {} of argument of function {}", c1_col_column->getName(), getName()),

ErrorCodes::ILLEGAL_COLUMN);

SeaRise · 2022-10-31T10:08:01Z

dbms/src/Functions/FunctionsString.cpp

+            const auto * substr_const_string = checkAndGetColumnConst<ColumnString>(c1_col_column.get());
+            if (!substr_const_string)
+            {
+                const auto * substr_vector_string = checkAndGetColumn<ColumnString>(c1_col_column.get());
+                const auto * str_const_string_value = checkAndGetColumn<ColumnString>(str_const_string->getDataColumnPtr().get());
+                constVector(str_const_string_value->getChars(), str_const_string_value->getOffsets(), substr_vector_string->getChars(), substr_vector_string->getOffsets(), val_num, res_vec);
+            }


Suggested change

const auto * substr_const_string = checkAndGetColumnConst<ColumnString>(c1_col_column.get());

if (!substr_const_string)

{

const auto * substr_vector_string = checkAndGetColumn<ColumnString>(c1_col_column.get());

const auto * str_const_string_value = checkAndGetColumn<ColumnString>(str_const_string->getDataColumnPtr().get());

constVector(str_const_string_value->getChars(), str_const_string_value->getOffsets(), substr_vector_string->getChars(), substr_vector_string->getOffsets(), val_num, res_vec);

}

if (const auto * substr_vector_string = checkAndGetColumn<ColumnString>(c1_col_column.get()))

{

const auto * str_const_string_value = checkAndGetColumn<ColumnString>(str_const_string->getDataColumnPtr().get());

constVector(str_const_string_value->getChars(), str_const_string_value->getOffsets(), substr_vector_string->getChars(), substr_vector_string->getOffsets(), val_num, res_vec);

}

else

throw Exception(

fmt::format("Illegal column {} of argument of function {}", c1_col_column->getName(), getName()),

ErrorCodes::ILLEGAL_COLUMN);

SeaRise · 2022-10-31T10:41:38Z

dbms/src/Functions/FunctionsString.cpp

+    }
+
+private:
+    static Int64 find(


It looks like we can use Volnitsky directly.
#include <Common/Volnitsky.h>
Ref

tiflash/dbms/src/Functions/FunctionsString.cpp

Line 4635 in 10ebe6b

class FunctionSubStringIndex : public IFunction

lizhenhuan and others added 20 commits June 25, 2022 23:12

Merge pull request #1 from pingcap/master

b2f0af8

Merge from pingcap repo

Implement ReverseUTF8/Reverse function push down pingcap#5111

28ae999

format

f02759a

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

Merge pull request #2 from pingcap/master

f1e7768

Merge from pingcap repo

implement reverse push down to tiflash

1e5be6d

pushdown reverse to TiFlash

306aa06

pushdown reverse to TiFlash

cfa9cad

push down reverse to tiflash

1ca32f3

reverse test case is ok

ba825b1

fix reverse test problem

9f2defe

fix reverse test case problem

062ba64

fix reverse test case problem

1a48b5f

Merge branch 'master' into master

d561f59

Merge branch 'master' into master

fad4933

Merge branch 'master' into master

6741072

merge form master

4ebb91b

Merge branch 'pingcap:master' into master

d72e330

implement instr push down to tiflash

ad3da65

Signed-off-by: lizhenhuan <1916038084@qq.com>

add instr

a0be732

Signed-off-by: lizhenhuan <1916038084@qq.com>

fix

324355a

Signed-off-by: lizhenhuan <1916038084@qq.com>

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 6, 2022

fix format error

4ea338e

Signed-off-by: lizhenhuan <1916038084@qq.com>

lizhenhuan added 2 commits October 8, 2022 18:56

fix

a136b00

Signed-off-by: lizhenhuan <1916038084@qq.com>

fix

90d1bd0

Signed-off-by: lizhenhuan <1916038084@qq.com>

xzhangxian1008 reviewed Oct 12, 2022

View reviewed changes

dbms/src/Functions/tests/gtest_strings_instr.cpp Show resolved Hide resolved

dbms/src/Functions/tests/gtest_strings_instr.cpp Show resolved Hide resolved

tests/fullstack-test/expr/instr.test Show resolved Hide resolved

tests/fullstack-test/expr/instr_utf8.test Show resolved Hide resolved

fix code

faa3bbf

Signed-off-by: lizhenhuan <1916038084@qq.com>

fix code

aaa6079

Signed-off-by: lizhenhuan <1916038084@qq.com>

fix code

7a0cc91

Signed-off-by: lizhenhuan <1916038084@qq.com>

xzhangxian1008 reviewed Oct 27, 2022

View reviewed changes

lizhenhuan added 2 commits October 29, 2022 21:56

fix code

360fb7c

Signed-off-by: lizhenhuan <1916038084@qq.com>

fix code

7ee7e26

Signed-off-by: lizhenhuan <1916038084@qq.com>

Merge branch 'master' into instr_push_down

779d481

ywqzzy approved these changes Oct 31, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 31, 2022

lizhenhuan added 2 commits October 31, 2022 14:08

remove const const

c412cb3

Signed-off-by: lizhenhuan <1916038084@qq.com>

Merge branch 'instr_push_down' of github.com:lizhenhuan/tiflash into …

f561a54

…instr_push_down

SeaRise reviewed Oct 31, 2022

View reviewed changes

dbms/src/Functions/FunctionsString.cpp Outdated Show resolved Hide resolved

dbms/src/Functions/FunctionsString.cpp Outdated Show resolved Hide resolved

dbms/src/Functions/FunctionsString.cpp Outdated Show resolved Hide resolved

fix code

1711659

Signed-off-by: lizhenhuan <1916038084@qq.com>

SeaRise reviewed Oct 31, 2022

View reviewed changes

zanmato1984 mentioned this pull request Mar 16, 2023

Call For Participation: implement function push down #5092

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement push down instr function to tiflash #6081

Implement push down instr function to tiflash #6081

lizhenhuan commented Oct 6, 2022 •

edited

Loading

ti-chi-bot commented Oct 6, 2022 •

edited

Loading

lizhenhuan commented Oct 6, 2022

sre-bot commented Oct 6, 2022

lizhenhuan commented Oct 8, 2022

sre-bot commented Oct 8, 2022

lizhenhuan commented Oct 23, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 25, 2022

lizhenhuan commented Oct 25, 2022

lizhenhuan commented Oct 29, 2022

xzhangxian1008 commented Oct 31, 2022

lizhenhuan commented Oct 31, 2022

lizhenhuan commented Oct 31, 2022

lizhenhuan commented Oct 31, 2022

SeaRise Oct 31, 2022

SeaRise Oct 31, 2022

SeaRise Oct 31, 2022

Implement push down instr function to tiflash #6081

Are you sure you want to change the base?

Implement push down instr function to tiflash #6081

Conversation

lizhenhuan commented Oct 6, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Oct 6, 2022 • edited Loading

lizhenhuan commented Oct 6, 2022

sre-bot commented Oct 6, 2022

Coverage for changed files

Coverage summary

lizhenhuan commented Oct 8, 2022

sre-bot commented Oct 8, 2022

Coverage for changed files

Coverage summary

lizhenhuan commented Oct 23, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 24, 2022

lizhenhuan commented Oct 25, 2022

lizhenhuan commented Oct 25, 2022

lizhenhuan commented Oct 29, 2022

xzhangxian1008 commented Oct 31, 2022

lizhenhuan commented Oct 31, 2022

lizhenhuan commented Oct 31, 2022

lizhenhuan commented Oct 31, 2022

SeaRise Oct 31, 2022

Choose a reason for hiding this comment

SeaRise Oct 31, 2022

Choose a reason for hiding this comment

SeaRise Oct 31, 2022

Choose a reason for hiding this comment

lizhenhuan commented Oct 6, 2022 •

edited

Loading

ti-chi-bot commented Oct 6, 2022 •

edited

Loading