DRILL-8526: Hive Predicate Push Down for ORC and Parquet #2995

shfshihuafeng · 2025-06-26T10:16:45Z

DRILL-8526: Hive Predicate Push Down for ORC and Parquet

Description

Drill do not support filter push down for orc format.When a large amount of data is filtered out, Predicate PushDown can significantly improve the query performance of ORC format.

I have implemented this featur and supported the data types are as follows:
int
bigint,
INYINT,
SMALLINT,
double,
float,
BOOLEAN
date,
TIMESTAMP,
decimal
string,
varchar
char

Testing

Through comparative testing , ORC format with filter pushdown achieves nearly a 20x performance improvement over execution without pushdown on following sql.

1.Import TPCH data. In my test, the amount of Table lineitem was 6001215.
2.set "store.hive.optimize_scan_filter_pushdown =true" by the option on drill web
3. execute sql select * from hive.lineitem_o where L_ORDERKEY=1;

(1)store.hive.optimize_scan_filter_pushdown =false

(2)store.hive.optimize_scan_filter_pushdown =true

cgivre

Thank you for this PR! There are a few minor comments, but the biggest thing that we'll need to merge this are some unit tests. Can you please add some to confirm that this is working as you intend?

...rage-hive/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilter.java

cgivre · 2025-06-26T12:53:39Z

...ve/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilterBuilder.java

+    RuntimeException> {
+  private static final Logger logger = LoggerFactory.getLogger(HiveFilterBuilder.class);
+
+  final private HiveScan groupScan;


Nit: please be consistent with private final vs final private

exec/java-exec/src/main/resources/drill-module.conf

cgivre · 2025-06-26T12:58:33Z

Wow! This is a serious performance improvement. Nicely done! One other thing... I've found that the limit was also not pushed down in a lot of storage plugins. It is usually fairly easy to add. Would you consider checking to see if the limit pushdown is there and if not, add it as well?

shfshihuafeng · 2025-06-27T01:01:21Z

Thank you for this PR! There are a few minor comments, but the biggest thing that we'll need to merge this are some unit tests. Can you please add some to confirm that this is working as you i

Wow! This is a serious performance improvement. Nicely done! One other thing... I've found that the limit was also not pushed down in a lot of storage plugins. It is usually fairly easy to add. Would you consider checking to see if the limit pushdown is there and if not, add it as well?

Thank you for this PR! There are a few minor comments, but the biggest thing that we'll need to merge this are some unit tests. Can you please add some to confirm that this is working as you intend?

@cgivre I did some test for this feature, including data type testing, logical expression testing, and some special expressions.as following:

data types: Int,bigint,INYINT,SMALLINT,double,float,BOOLEAN,date,TIMESTAMP,decimal,string,varchar and char
logical expression testing: and ,or
some special expression: is null is false and is not true and so on

i will add some unit tests for it

shfshihuafeng · 2025-06-27T01:11:12Z

Thank you for this PR! There are a few minor comments, but the biggest thing that we'll need to merge this are some unit tests. Can you please add some to confirm that this is working as you intend?

contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java

...main/java/org/apache/drill/exec/store/hive/readers/filter/HiveCompareFunctionsProcessor.java

cgivre · 2025-06-27T01:49:05Z

...rage-hive/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilter.java

+import org.apache.hive.com.esotericsoftware.kryo.io.Output;
+
+public class HiveFilter {
+  private SearchArgument searchArgument;


Please make final.

...ve/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilterBuilder.java

cgivre

I submitted some minor changes. Just a reminder but we really need unit tests in order to merge this.

Also, have you considered adding a limit pushdown? It is usually pretty easy to do and only involves:

Implementing two methods in the group scan (HiveScan) which are: supportsLimitPushdown and applyLimit.
Passing the limit through the subscans.
Adding some logic in the readers to stop when the limit is reached.
Maybe it would be best to open a new JIRA for that, but IMHO, it is one of the easiest and most effective pushdowns that can be implemented yet Drill didn't seem to do for all the plugins.

...main/java/org/apache/drill/exec/store/hive/readers/filter/HiveCompareFunctionsProcessor.java

cgivre · 2025-06-30T14:29:13Z

...main/java/org/apache/drill/exec/store/hive/readers/filter/HiveCompareFunctionsProcessor.java

+    }
+    return false;
+  }
+}


Nit: Please add a blank line at the end of all classes. Here and elsewhere.

...rage-hive/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilter.java

cgivre · 2025-06-30T14:32:40Z

...ve/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilterBuilder.java

+  private SearchArgument.Builder builder;
+  private HashMap<String, SqlTypeName> dataTypeMap;
+
+  HiveFilterBuilder(LogicalExpression le, HashMap<String, SqlTypeName> dataTypeMap) {


Should this be public?

cgivre · 2025-06-30T14:33:01Z

...ve/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilterBuilder.java

+
+  private boolean allExpressionsConverted = false;
+  private SearchArgument.Builder builder;
+  private HashMap<String, SqlTypeName> dataTypeMap;


Please make any variables final which can be made final.

cgivre · 2025-06-30T14:37:13Z

...ve/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilterBuilder.java

+        logger.warn("Unsupported logical operator:{} for push down", functionName);
+        return builder;
+      }
+      for (int i = 0; i < args.size(); ++i) {


Can we replace with enhanced for-loop?

cgivre · 2025-06-30T14:37:45Z

...ve/core/src/main/java/org/apache/drill/exec/store/hive/readers/filter/HiveFilterBuilder.java

+  private PredicateLeaf.Type convertLeafType(SqlTypeName sqlTypeName) {
+    PredicateLeaf.Type type = null;
+    switch (sqlTypeName) {
+    case BOOLEAN:


Nit: Please indent switch statement.

shfshihuafeng · 2025-06-30T16:15:13Z

I submitted some minor changes. Just a reminder but we really need unit tests in order to merge this.

Also, have you considered adding a limit pushdown? It is usually pretty easy to do and only involves:

Implementing two methods in the group scan (HiveScan) which are: supportsLimitPushdown and applyLimit.

Passing the limit through the subscans.

Adding some logic in the readers to stop when the limit is reached.
Maybe it would be best to open a new JIRA for that, but IMHO, it is one of the easiest and most effective pushdowns that can be implemented yet Drill didn't seem to do for all the plugins.

@cgivre ok, I will add some unit tests and supoort limit push down later

shfshihuafeng · 2025-06-30T16:20:10Z

I submitted some minor changes. Just a reminder but we really need unit tests in order to merge this.
Also, have you considered adding a limit pushdown? It is usually pretty easy to do and only involves:

Implementing two methods in the group scan (HiveScan) which are: supportsLimitPushdown and applyLimit.

Passing the limit through the subscans.

Adding some logic in the readers to stop when the limit is reached.
Maybe it would be best to open a new JIRA for that, but IMHO, it is one of the easiest and most effective pushdowns that can be implemented yet Drill didn't seem to do for all the plugins.

@cgivre ok, I will add some unit tests and support limit pushdown

cgivre · 2025-06-30T16:48:22Z

I submitted some minor changes. Just a reminder but we really need unit tests in order to merge this.
Also, have you considered adding a limit pushdown? It is usually pretty easy to do and only involves:

Implementing two methods in the group scan (HiveScan) which are: supportsLimitPushdown and applyLimit.

Passing the limit through the subscans.

Adding some logic in the readers to stop when the limit is reached.
Maybe it would be best to open a new JIRA for that, but IMHO, it is one of the easiest and most effective pushdowns that can be implemented yet Drill didn't seem to do for all the plugins.

@cgivre ok, I will add some unit tests and support limit pushdown

@shfshihuafeng Just to be clear, we can't merge this without unit tests.

You don't have to do the limit pushdown if you don't want to or aren't able to, but I just wanted to recommend it as it can be a big performance benefit for not much work. Alternatively, you can create a separate pull request for that and do it later. It is your choice..

shfshihuafeng · 2025-07-01T14:02:41Z

I submitted some minor changes. Just a reminder but we really need unit tests in order to merge this.

Also, have you considered adding a limit pushdown? It is usually pretty easy to do and only involves:

Implementing two methods in the group scan (HiveScan) which are: supportsLimitPushdown and applyLimit.

Passing the limit through the subscans.

Adding some logic in the readers to stop when the limit is reached.
Maybe it would be best to open a new JIRA for that, but IMHO, it is one of the easiest and most effective pushdowns that can be implemented yet Drill didn't seem to do for all the plugins.

@cgivre I think it is best to open a new JIRA,so you can refer to following link for hive limit push down,and i commit pr for this
JIRA: https://issues.apache.org/jira/browse/DRILL-8527
@cgivre i am adding unit tests for Drill-8526. It may take some time

shfshihuafeng · 2025-07-19T15:55:47Z

@cgivre I added unit tests which were not run on JDK 11+. Therefore, I submitted another PR in version drill 18 to run these unit tests on jdk 8.

cgivre

LGTM +1 Unless there are any objections, I think we should merge this. It doesn't break anything and apparently the Hive unit tests were not running (and already broken).

@shfshihuafeng It would be nice if you could fix them and assist with updating the Hive client to a newer version, but since this is adding value for you, I'd like to get it merged.

cgivre requested changes Jun 26, 2025

View reviewed changes

cgivre assigned shfshihuafeng Jun 26, 2025

cgivre added enhancement PRs that add a new functionality to Drill doc-impacting PRs that affect the documentation performance PRs that Improve Performance labels Jun 26, 2025

shfshihuafeng closed this Jun 27, 2025

shfshihuafeng reopened this Jun 27, 2025

cgivre reviewed Jun 27, 2025

View reviewed changes

contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java Outdated Show resolved Hide resolved

cgivre requested changes Jun 27, 2025

View reviewed changes

shfshihuafeng force-pushed the hive_orc_pushdown branch 2 times, most recently from 019ce0d to 0d20e3d Compare June 27, 2025 05:21

shfshihuafeng requested a review from cgivre June 28, 2025 01:47

cgivre requested changes Jun 30, 2025

View reviewed changes

shfshihuafeng force-pushed the hive_orc_pushdown branch 3 times, most recently from f1c632f to f3ef7a7 Compare July 19, 2025 15:44

shfshihuafeng requested a review from cgivre July 19, 2025 16:02

DRILL-8526: Hive Predicate Push Down for ORC and Parquet

692a81b

shfshihuafeng force-pushed the hive_orc_pushdown branch from ee1ea37 to 692a81b Compare July 20, 2025 03:11

cgivre mentioned this pull request Jul 23, 2025

Just to test orc push down on JDK8 #3001

Closed

cgivre approved these changes Aug 7, 2025

View reviewed changes

cgivre merged commit 6cea55d into apache:master Aug 7, 2025
7 checks passed

DRILL-8526: Hive Predicate Push Down for ORC and Parquet #2995

DRILL-8526: Hive Predicate Push Down for ORC and Parquet #2995

Conversation

shfshihuafeng commented Jun 26, 2025

DRILL-8526: Hive Predicate Push Down for ORC and Parquet

Description

Testing

Uh oh!

cgivre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cgivre Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cgivre commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shfshihuafeng commented Jun 27, 2025

Uh oh!

shfshihuafeng commented Jun 27, 2025

Uh oh!

Uh oh!

Uh oh!

cgivre Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cgivre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cgivre Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cgivre Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

cgivre Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

cgivre Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

cgivre Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

shfshihuafeng commented Jun 30, 2025

Uh oh!

shfshihuafeng commented Jun 30, 2025

Uh oh!

cgivre commented Jun 30, 2025

Uh oh!

shfshihuafeng commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shfshihuafeng commented Jul 19, 2025

Uh oh!

cgivre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cgivre commented Jun 26, 2025 •

edited

Loading

shfshihuafeng commented Jul 1, 2025 •

edited

Loading