[Backport 2.19-dev] Core Implementation of rex Command In PPL
#4239
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ject#4109) * rex - initial implementation Signed-off-by: Jialiang Liang <jiallian@amazon.com> * stop using utils Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix spotless check Signed-off-by: Jialiang Liang <jiallian@amazon.com> * offset_field - initial implementation Signed-off-by: Jialiang Liang <jiallian@amazon.com> * max_match - initial implementation Signed-off-by: Jialiang Liang <jiallian@amazon.com> * sed - initial implementation Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix name capture group for extraction Signed-off-by: Jialiang Liang <jiallian@amazon.com> * add rex rst doc Signed-off-by: Jialiang Liang <jiallian@amazon.com> * IT - initial setup Signed-off-by: Jialiang Liang <jiallian@amazon.com> * add a analyzer test for legacy engine Signed-off-by: Jialiang Liang <jiallian@amazon.com> * Add UT for rex Signed-off-by: Jialiang Liang <jiallian@amazon.com> * sed - add pushdown for sed and explain IT and IT with fix Signed-off-by: Jialiang Liang <jiallian@amazon.com> * anonymizer - add rex for anonymizer and test Signed-off-by: Jialiang Liang <jiallian@amazon.com> * Add cross cluster IT for rex Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - resolve comments for rst doc 0 Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - address some comments 1 Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - resolve comment in rst doc to add a java doc link Signed-off-by: Jialiang Liang <jiallian@amazon.com> * kai - modify the bin ast builder test Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - fix the extraction behavior without filter even when there is zero match Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix rex explain no pushdown Signed-off-by: Jialiang Liang <jiallian@amazon.com> * change the offset val output format Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix rst file Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - SWITCH TO USE CALCITE NATIVE OPERATORS Signed-off-by: Jialiang Liang <jiallian@amazon.com> * Peng - fix tests after operator change Signed-off-by: Jialiang Liang <jiallian@amazon.com> * support mode=extract and update doc Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix the issue after rebase Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - enforce specifying field in antlr for now Signed-off-by: Jialiang Liang <jiallian@amazon.com> * relocate rex cmd IT Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - simplify vistFunciton Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - add UT for RexExtractMultiFunction Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - add UT RexOffsetFunction Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix some tests Signed-off-by: Jialiang Liang <jiallian@amazon.com> * DECOUPLE SED + OFFSET FIELD Signed-off-by: Jialiang Liang <jiallian@amazon.com> * Improve error handling for extract Signed-off-by: Jialiang Liang <jiallian@amazon.com> * add this rex rst into index Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix return type in extract multi Signed-off-by: Jialiang Liang <jiallian@amazon.com> * add rex doc into doc test Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix doc test Signed-off-by: Jialiang Liang <jiallian@amazon.com> * Fix linting Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix rebase issue Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix regex anonymizer tests Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix analyzer test and setup to use util function Signed-off-by: Jialiang Liang <jiallian@amazon.com> * lint fix Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix doc test Signed-off-by: Jialiang Liang <jiallian@amazon.com> * Add max match limit implementation Signed-off-by: Jialiang Liang <jiallian@amazon.com> * fix anonymizer test Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - simplify if Signed-off-by: Jialiang Liang <jiallian@amazon.com> * peng - make extract multi to only handle the case of max_match > 1 Signed-off-by: Jialiang Liang <jiallian@amazon.com> --------- Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
requested review from
GumpacG,
LantaoJin,
MaxKsyunz,
Swiddis,
YANG-DB,
Yury-Fridlyand,
acarbonetto,
anirudha,
dai-chen,
derek-ho,
forestmvey,
joshuali925,
kavithacm,
mengweieric,
noCharger,
penghuo,
ps48,
qianheng-aws,
seankao-az and
ykmr1224
as code owners
ykmr1224
previously approved these changes
Sep 5, 2025
|
I saw the doc tests failed because the sample data used for doctest is different than the one we used on main. |
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
dai-chen
approved these changes
Sep 9, 2025
Swiddis
approved these changes
Sep 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Manual backport
Related Issues
rexCommand In PPL #4109Check List
--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.