Skip to content

[OPPRO-104] Support more cases of filter and its pushdown #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jun 6, 2022
Merged

[OPPRO-104] Support more cases of filter and its pushdown #14

merged 6 commits into from
Jun 6, 2022

Conversation

@rui-mo
Copy link
Collaborator

@rui-mo rui-mo commented May 20, 2022 •
edited
Loading

Supported below filter and project expressions:

EndsWith, EqualTo, Or, Contains, In, Not, Like, StartsWith, Substring, Switch (If).

Added logic to separate conditions into two parts. One part can be pushed down, and another part should be handled by remaining filter in TableScan.

Added a quick implementation for IN of double type. The double value will be cast into bigint.

Supported different types for filter push down, including int/bigint/string.

Exposed the registration for min/max.

Use the pointer of memory pool.

@rui-mo rui-mo mentioned this pull request May 20, 2022
Merged
@rui-mo rui-mo mentioned this pull request May 31, 2022
Merged
@rui-mo
Copy link
Collaborator Author

rui-mo commented Jun 1, 2022

@marin-ma @JkSelf Could you help take a review or test? Thanks.

@rui-mo
Copy link
Collaborator Author

rui-mo commented Jun 6, 2022

Merging this PR. If any comment, you can still comment here.

@rui-mo rui-mo merged commit 361ae8f into oap-project:main Jun 6, 2022
zhejiangxiaomai pushed a commit that referenced this pull request Jun 9, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
715626c
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jun 13, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
0be7f93
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jun 16, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
dd25c47
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jun 21, 2022
@rui-mo
[OPPRO-104] Support more cases of filter and its pushdown (#14)
8c00d3a
zhejiangxiaomai pushed a commit that referenced this pull request Jun 21, 2022
@rui-mo
[OPPRO-104] Support more cases of filter and its pushdown (#14)
3634b67
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jun 29, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
bffe431
zhejiangxiaomai pushed a commit that referenced this pull request Jul 12, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
419a05a
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jul 26, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
01cede2
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Aug 9, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
458badd
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Aug 17, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
f55b7ee
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Aug 18, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
8ca35c9
zhejiangxiaomai pushed a commit that referenced this pull request Aug 24, 2022
@rui-mo
[OPPRO-104] Support more cases of filter and its pushdown (#14)
a82b9a3
zhejiangxiaomai pushed a commit that referenced this pull request Sep 1, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
6209262
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Sep 16, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
ff9d3f9
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Sep 22, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
6e52749
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Oct 20, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
5e597cf
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Oct 20, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
2b36f2b
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Dec 15, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
9043f99
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Dec 15, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
c5698eb
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Dec 21, 2022
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
649ea42
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jan 3, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
17d8dab
zhejiangxiaomai pushed a commit that referenced this pull request Jan 11, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
3f77eec
zhejiangxiaomai pushed a commit that referenced this pull request Jan 11, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
d895ba0
zhejiangxiaomai pushed a commit that referenced this pull request Jan 31, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
13040ff
zhejiangxiaomai pushed a commit that referenced this pull request Feb 22, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
d5dd05b
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Feb 27, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
c11717d
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Mar 6, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
81de8f0
jinchengchenghh pushed a commit to jinchengchenghh/velox that referenced this pull request Mar 22, 2023
@liujiayi771
sparksql decimal avg add some check (oap-project#14)
a91568a
jinchengchenghh pushed a commit to jinchengchenghh/velox that referenced this pull request Mar 23, 2023
@liujiayi771 @JkSelf
sparksql decimal avg add some check (oap-project#14)
9071aa2
jinchengchenghh pushed a commit to jinchengchenghh/velox that referenced this pull request Mar 24, 2023
@liujiayi771 @JkSelf
sparksql decimal avg add some check (oap-project#14)
0ae6251
jinchengchenghh pushed a commit to jinchengchenghh/velox that referenced this pull request Mar 27, 2023
@liujiayi771 @JkSelf
sparksql decimal avg add some check (oap-project#14)
c85a76d
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Mar 27, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
ef9a6d0
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Mar 29, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
a6b441f
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Mar 29, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
8239a92
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Apr 14, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
ff5ba92
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Apr 17, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
4fd2867
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Apr 19, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
fc43122
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Apr 20, 2023
@rui-mo @zhejiangxiaomai
[OPPRO-104] Support more cases of filter and its pushdown (#14)
3499429
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox May 31, 2023
@zhejiangxiaomai
Folder: substrait
efc0949 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox May 31, 2023
@zhejiangxiaomai
Folder: substrait
d8ad570 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox May 31, 2023
@zhejiangxiaomai
Folder: substrait
f5e31fa 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jul 3, 2023
@zhejiangxiaomai
Folder: substrait
e94db80 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai referenced this pull request in zhejiangxiaomai/velox Jul 4, 2023
@zhejiangxiaomai
Folder: substrait
6ae5844 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhztheplayer pushed a commit to zhztheplayer/velox that referenced this pull request Jul 27, 2023
@zhejiangxiaomai
Folder: substrait
20f54b6 
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown oap-project#14
marin-ma pushed a commit to marin-ma/velox-oap that referenced this pull request Dec 15, 2023
@rui-mo
[GJ-13] refine docs (oap-project#14)
fa2a4ec 
* update docs

* refine docs
@rui-mo rui-mo deleted the wip_filter branch February 20, 2024 03:15
prestodb-ci pushed a commit that referenced this pull request Jun 24, 2025
@facebook-github-bot
fix(vector): VectorMaker-inl.h: fix llvm-19-exposed UBSAN signed-inte…
5108202 
…ger-overflow (facebookincubator#13831)

Summary:
Pull Request resolved: facebookincubator#13831

This avoids the following errors:

```
fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_abs.h:56:41: runtime error: negation of -9223372036854775808 cannot be represented in type 'long'; cast to an unsigned type to negate this value to itself
    #0 0x000000346ce5 in std::abs(long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_abs.h:56
    #1 0x000000345879 in std::shared_ptr<facebook::velox::BiasVector<facebook::velox::test::EvalTypeHelper<long>::Type>> facebook::velox::test::VectorMaker::biasVector<long>(std::vector<std::optional<long>, std::allocator<std::optional<long>>> const&) fbcode/velox/vector/tests/utils/VectorMaker-inl.h:58
    #2 0x000000344d34 in facebook::velox::test::BiasVectorErrorTest::errorTest(std::vector<std::optional<long>, std::allocator<std::optional<long>>>) fbcode/velox/vector/tests/BiasVectorTest.cpp:39
    #3 0x00000033ec99 in facebook::velox::test::BiasVectorErrorTest_checkRangeTooLargeError_Test::TestBody() fbcode/velox/vector/tests/BiasVectorTest.cpp:44
    #4 0x7fe0a2342c46 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2727
    #5 0x7fe0a234275d in testing::Test::Run() fbsource/src/gtest.cc:2744
    #6 0x7fe0a2345fb3 in testing::TestInfo::Run() fbsource/src/gtest.cc:2890
    #7 0x7fe0a234c8eb in testing::TestSuite::Run() fbsource/src/gtest.cc:3068
    #8 0x7fe0a237b52b in testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:6059
    #9 0x7fe0a237a0a2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2727
    #10 0x7fe0a23797f5 in testing::UnitTest::Run() fbsource/src/gtest.cc:5599
    #11 0x7fe0a2239800 in RUN_ALL_TESTS() fbsource/gtest/gtest.h:2334
    #12 0x7fe0a223952c in main fbcode/common/gtest/LightMain.cpp:20
    #13 0x7fe09ec2c656 in __libc_start_call_main /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #14 0x7fe09ec2c717 in __libc_start_main@GLIBC_2.2.5 /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../csu/libc-start.c:409:3
    #15 0x00000033d8b0 in _start /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116

UndefinedBehaviorSanitizer: signed-integer-overflow fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_abs.h:56:41
```
Avoid overflow by using the expression (static_cast<uint64_t>(1) + ~static_cast<uint64_t>(min)) to calculate the absolute value of min without using std::abs

Reviewed By: dmm-fb, peterenescu

Differential Revision: D76901449

fbshipit-source-id: 7eb3bd0f83e42f44cdf34ea1759f3aa9e1042dae
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rui-mo