Support Regex for replace eval function

[ahkcs]

Description
Enhances the replace() function to support regular expressions, enabling advanced text manipulation in PPL queries.

---

Examples

Before (literal only):

source=logs | eval cleaned = replace(message, "ERROR", "WARN")

After (full regex):

# Remove digits
source=logs | eval cleaned = replace(message, '\d+', '')

# Swap date format (MM/DD/YYYY → DD/MM/YYYY)
source=logs | eval reformatted = replace(date, '^(\d{2})/(\d{2})/(\d{4})$', '\2/\1/\3')

# Extract domain from email
source=users | eval domain = replace(email, '.*@(.+)', '\1')

[RyanL1997]

Hi @ahkcs , thanks for the change. I just left a comment of re-using existing operator of regex replacement and you can take a look. If there is any questions, feel free to ask me.

[RyanL1997]

Instead of creating the new UDF we should re-use the existing supported sql opperator. You can reference to my previous implementation at sed in rex that I was doing the same thing:

• [Feature] Implementation of mode sed and offset_field in rex PPL command #4241

For more details you can also reference to: #4109 (comment)

REX_SED - Complete native Calcite optimization
REGEXP_REPLACE_PG_3 for basic substitution
REGEXP_REPLACE_PG_4 for flagged substitution (g, i)
REGEXP_REPLACE_5 for nth occurrence
TRANSLATE3 for transliteration (y/from/to/)

As you can see there are various of options that available for regex replace and I think you can choose the one you need.

[RyanL1997]

More specifically, you may need to use this:

BuiltinFunctionName.INTERNAL_REGEXP_REPLACE_3,

[ahkcs]

Hi @RyanL1997 , thanks for the suggestion, I have updated the implementation to remove the custom UDF and leverage the existing sql operator

[RyanL1997]

@ahkcs , also I was wondering for the usage of the regex pattern in this function, will it support captured name group? If that is the case, there are some known limitation of java regex library. see details at #4434 (comment), and lets see if we need to call this out in the related docs.

[ahkcs]

@ahkcs , also I was wondering for the usage of the regex pattern in this function, will it support captured name group? If that is the case, there are some known limitation of java regex library. see details at #4434 (comment), and lets say if we need to call this out in the related docs.

@RyanL1997 , thanks for calling it out, named groups like (?<domain_name>.+) are used by the rex command, which does have the Java regex limitation you mentioned (no underscores allowed in group names). The replace() function avoids this limitation entirely by using numbered references.

For example:

• Rex fails: rex field=email ".+@(?<domain_name>.+)" (underscore not allowed)
• Replace works: eval domain_name = replace(email, '.+@(.+)', '\1') (field name defined in eval, not in regex)

So the Java named group limitations in #4434 don't apply to replace() function. The field naming is controlled by the eval command, bypassing Java identifier restrictions.

[RyanL1997]

@ahkcs , also I was wondering for the usage of the regex pattern in this function, will it support captured name group? If that is the case, there are some known limitation of java regex library. see details at #4434 (comment), and lets say if we need to call this out in the related docs.

@RyanL1997 , thanks for calling it out, named groups like (?<domain_name>.+) are used by the rex command, which does have the Java regex limitation you mentioned (no underscores allowed in group names). The replace() function avoids this limitation entirely by using numbered references.

For example:

• Rex fails: rex field=email ".+@(?<domain_name>.+)" (underscore not allowed)
• Replace works: eval domain_name = replace(email, '.+@(.+)', '\1') (field name defined in eval, not in regex)

So the Java named group limitations in #4434 don't apply to replace() function. The field naming is controlled by the eval command, bypassing Java identifier restrictions.

Make sense, since in the case of this PR, the extraction should be done by eval so it won't triggered the java regex pattern limitation for named capture group.

[RyanL1997]

Hi @ahkcs , thanks for the update of the implementation and I just left some comments.

[RyanL1997]

Is this because the requirement requires java style? What is the expected behavior over here?

[ahkcs]

Because we use java regex internally but we want to support PCRE style syntax, so we add this conversion logic to translate PCRE syntax to java regex

[RyanL1997]

non-blocking: I remember I do have the same logic for sed, maybe we can extract them into a certain util.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4456-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a8f08ad3e101e05aefecca4765d484d1992a12b3
# Push it to GitHub
git push --set-upstream origin backport/backport-4456-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4456-to-2.19-dev.