[Feature] support regex_instr #36958

liuzhixing1006 · 2023-12-13T08:32:31Z

Why I'm doing:

What I'm doing:

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

CLAassistant · 2023-12-13T08:32:40Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Conflicts: be/test/exprs/string_fn_test.cpp

liuzhixing1006 · 2023-12-13T09:12:38Z

This resolves #19616

fzhedu · 2023-12-13T09:31:55Z

docs/en/sql-reference/sql-functions/like-predicate-functions/regexp_instr.md

+MySQL > SELECT regexp_instr('abCdE', 'ABC');
+--------------------------------------------------------+
+| regexp_instr('abCdE', 'AbC');                          |
+--------------------------------------------------------+


why change 'ABC' to 'AbC' ?

fzhedu · 2023-12-13T09:33:24Z

be/test/exprs/string_fn_test.cpp

+    std::string strs[] = {"Alex test", "   Alexx test", "AAAAAlex test", "AAlex Alex", "adssdaAlexxx"};
+    std::string res[] = {"1", "4", "5", "2", "7"};
+
+    for (int i = 0; i < sizeof(strs) / sizeof(strs[0]); ++i) {


why not strs.size()?

fzhedu · 2023-12-13T09:34:23Z

be/test/exprs/string_fn_test.cpp

+    std::string strs[] = {"Alex test", "   Alexx test", "AAAAAlex test", "AAlex Alex", "adssdaAlexxx"};
+    std::string res[] = {"1", "4", "5", "2", "7"};


what if str or pattern are null or empty string?

fzhedu · 2023-12-13T09:38:28Z

docs/en/sql-reference/sql-functions/like-predicate-functions/regexp_instr.md

+## Syntax
+
+```Haskell
+VARCHAR regexp_instr(VARCHAR str, VARCHAR pattern)


return int or bigint? what's the range?

fzhedu · 2023-12-13T09:39:13Z

docs/en/sql-reference/sql-functions/like-predicate-functions/regexp_instr.md

+
+## Description
+This function returns the first matching index in the target value which matches the regular expression pattern. The pattern must completely match some parts of str so that the function can return the exactly index. Be careful, returned character index begin at 1. If no matches are found, it will return an 0.
+


add a note how to process null. keep the same behavior with other DB

fzhedu · 2023-12-13T09:40:42Z

be/src/exprs/string_functions.cpp

+        if (content_viewer.is_null(row)) {
+            result.append(0);
+            continue;


why not return null according to null's semantics, what the behavior of other DB?

Mysql function docs: https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-instr

regexp-instr returns the starting index of the substring of the string expr that matches the regular expression specified by the pattern pat, 0 if there is no match. If expr or pat is NULL, the return value is NULL. Character indexes begin at 1.

When row_data equals "", any pattern can't match row_data, so the function will return 0 instead null's semantics.

fzhedu · 2023-12-13T09:43:26Z

be/src/exprs/string_functions.cpp

+    if (state->const_pattern) {
+        return regexp_instr_const(state, columns);
+    }


what if strings are const?

satanson · 2023-12-13T09:34:10Z

gensrc/script/functions.py

@@ -300,6 +300,8 @@
     'StringFunctions::regexp_extract_prepare', 'StringFunctions::regexp_close'],
    [30321, 'regexp_extract_all', 'ARRAY_VARCHAR', ['VARCHAR', 'VARCHAR', 'BIGINT'], 'StringFunctions::regexp_extract_all',
     'StringFunctions::regexp_extract_prepare', 'StringFunctions::regexp_close'],
+    [30322, 'regexp_instr', 'INT', ['VARCHAR', 'VARCHAR'], 'StringFunctions::regexp_instr',
+     'StringFunctions::regexp_instr_prepare', 'StringFunctions::regexp_close'],
    [30330, 'regexp_replace', 'VARCHAR', ['VARCHAR', 'VARCHAR', 'VARCHAR'], 'StringFunctions::regexp_replace',


re-range the functions in the ascend order of function ID(s)

satanson · 2023-12-13T09:35:39Z

be/test/exprs/string_fn_test.cpp

@@ -3546,6 +3546,217 @@ PARALLEL_TEST(VecStringFunctionsTest, regexpExtractAllConst) {
    }
 }

+PARALLEL_TEST(VecStringFunctionsTest, regexpInstrPattern) {


add some tests on real utf8 string?

satanson · 2023-12-13T09:40:29Z

be/src/exprs/string_functions.cpp

+        context->set_error(error.str().c_str());
+        return Status::InvalidArgument(error.str());
+    }
+


Take into consideration that the zeroth argument is also constant. for an example

select regex_instr('abcdef', 'ab[cd]*') from a_very_large_table

or implement constant folding of this function is ScalarOperatorFunctions.java

satanson · 2023-12-13T09:50:10Z

be/src/exprs/string_functions.cpp

+    auto size = columns[0]->size();
+    ColumnBuilder<TYPE_INT> result(size);
+    for (int row = 0; row < size; ++row) {
+        if (pattern_str.empty()) {


when pattern_str.empty() is true, can we just return a constant result column instead of processing each row inside loop.

satanson · 2023-12-13T09:52:15Z

be/src/exprs/string_functions.cpp

+        std::string row_str = row_value.get_data();
+        std::string match_str = match.data();
+
+        auto index = row_str.find(match_str);


suggest to use instr function instead of find, the former outperforms the the latter.

silverbullet233 · 2023-12-13T11:07:48Z

docs/zh/sql-reference/sql-functions/like-predicate-functions/regexp_instr.md

+displayed_sidebar: "Chinese"
+---
+
+# regexp_extract


function name is mismatched, need to be fixed

silverbullet233 · 2023-12-13T11:07:58Z

docs/zh/sql-reference/sql-functions/like-predicate-functions/regexp_instr.md

+## 语法
+
+```Haskell
+regexp_extract(str, pattern)


same as above

…gexp_instr Conflicts: be/src/exprs/string_functions.cpp

liuzhixing1006 added 5 commits December 9, 2023 22:54

FE Commit: support regexp_instr function

5ccac3d

BE Commit: support regexp_instr function

a979d74

BE

bca5f4b

be commit: Add doc & Update some response body

834ff78

BE Commit: add CN-doc

1f9e536

liuzhixing1006 requested a review from a team as a code owner December 13, 2023 08:32

github-actions bot added the documentation Improvements or additions to documentation label Dec 13, 2023

mergify bot assigned liuzhixing1006 Dec 13, 2023

liuzhixing1006 mentioned this pull request Dec 13, 2023

[function]regexp_instr #19616

Closed

liuzhixing1006 changed the title ~~[Feature] support regex_instr #19616~~ [Feature] support regex_instr Dec 13, 2023

Merge remote-tracking branch 'upstream/main' into regexp_instr

a62a2b6

Conflicts: be/test/exprs/string_fn_test.cpp

fzhedu reviewed Dec 13, 2023

View reviewed changes

satanson reviewed Dec 13, 2023

View reviewed changes

silverbullet233 reviewed Dec 13, 2023

View reviewed changes

liuzhixing1006 added 4 commits December 14, 2023 02:01

BE commit：update docs and fix INT boundary problem

d7b069b

BE Commit：fix const column compare & Utf8 mode

a239efb

Merge branch 'main' of https://github.com/StarRocks/starrocks into re…

29494b8

…gexp_instr Conflicts: be/src/exprs/string_functions.cpp

BE Commit: State add row

d6d7e76

liuzhixing1006 requested review from satanson, silverbullet233 and fzhedu December 19, 2023 14:20

BE Commit: modify doc

2e2b2b5

fzhedu approved these changes Dec 29, 2023

View reviewed changes

DanRoscigno removed the documentation Improvements or additions to documentation label Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] support regex_instr #36958

[Feature] support regex_instr #36958

liuzhixing1006 commented Dec 13, 2023 •

edited by github-actions bot

Loading

CLAassistant commented Dec 13, 2023

liuzhixing1006 commented Dec 13, 2023

fzhedu Dec 13, 2023

fzhedu Dec 13, 2023

fzhedu Dec 13, 2023

fzhedu Dec 13, 2023

fzhedu Dec 13, 2023

fzhedu Dec 13, 2023

liuzhixing1006 Dec 13, 2023

fzhedu Dec 13, 2023

satanson Dec 13, 2023

satanson Dec 13, 2023

satanson Dec 13, 2023

satanson Dec 13, 2023

satanson Dec 13, 2023

silverbullet233 Dec 13, 2023

silverbullet233 Dec 13, 2023

		std::string strs[] = {"Alex test", " Alexx test", "AAAAAlex test", "AAlex Alex", "adssdaAlexxx"};
		std::string res[] = {"1", "4", "5", "2", "7"};


		## Description
		This function returns the first matching index in the target value which matches the regular expression pattern. The pattern must completely match some parts of str so that the function can return the exactly index. Be careful, returned character index begin at 1. If no matches are found, it will return an 0.

[Feature] support regex_instr #36958

Are you sure you want to change the base?

[Feature] support regex_instr #36958

Conversation

liuzhixing1006 commented Dec 13, 2023 • edited by github-actions bot Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

CLAassistant commented Dec 13, 2023

liuzhixing1006 commented Dec 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuzhixing1006 commented Dec 13, 2023 •

edited by github-actions bot

Loading