[SPARK-12639] [SQL] Mark Filters Fully Handled By Sources with * #11317

RussellSpitzer · 2016-02-23T02:30:06Z

What changes were proposed in this pull request?

In order to make it clear which filters are fully handled by the
underlying datasource we will mark them with an *. This will give a
clear visual queue to users that the filter is being treated differently
by catalyst than filters which are just presented to the underlying
DataSource.

Examples from the FilteredScanSuite, in this example c IN (...) is handled by the source, b < ... is not

Before

//SELECT a FROM oneToTenFiltered WHERE a + b > 9 AND b < 16 AND c IN ('bbbbbBBBBB', 'cccccCCCCC', 'dddddDDDDD', 'foo') 
== Physical Plan ==
Project [a#0]
+- Filter (((a#0 + b#1) > 9) && (b#1 < 16))
   +- Scan SimpleFilteredScan(1,10)[a#0,b#1] PushedFilters: [LessThan(b,16), In(c, [bbbbbBBBBB,cccccCCCCC,dddddDDDDD,foo]]

After

== Physical Plan ==
Project [a#0]
+- Filter (((a#0 + b#1) > 9) && (b#1 < 16))
   +- Scan SimpleFilteredScan(1,10)[a#0,b#1] PushedFilters: [LessThan(b,16), *In(c, [bbbbbBBBBB,cccccCCCCC,dddddDDDDD,foo]]

How was the this patch tested?

Manually tested with the Spark Cassandra Connector, a source which fully handles underlying filters. Now fully handled filters appear with an * next to their names. I can add an automated test as well if requested

Post 1.6.1
Tested by modifying the FilteredScanSuite to run explains.

rxin · 2016-02-23T05:13:31Z

cc @yhuai

yhuai · 2016-02-23T18:34:31Z

ok to test

yhuai · 2016-02-23T18:35:58Z

@RussellSpitzer If you get a chance, can you add an example in the description showing the plan before and after this change?

yhuai · 2016-02-23T18:42:34Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

   */
  protected[sql] def selectFilters(
    relation: BaseRelation,
-    predicates: Seq[Expression]): (Seq[Expression], Seq[Filter]) = {
+    predicates: Seq[Expression]): (Seq[Expression], Seq[Filter], Seq[Filter]) = {


Maybe (Seq[Expression], Seq[Filter], Set[Filter])?

yhuai · 2016-02-23T18:42:53Z

Thank you @RussellSpitzer! It looks good. Only one comment.

SparkQA · 2016-02-23T19:00:42Z

Test build #51781 has finished for PR 11317 at commit c33daae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2016-03-16T23:12:44Z

ok to test

SparkQA · 2016-03-16T23:34:09Z

Test build #53374 has finished for PR 11317 at commit c33daae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-05-07T03:13:25Z

@RussellSpitzer I saw you answered my ping before. Excuse my ping here again.

RussellSpitzer · 2016-05-10T00:45:21Z

@HyukjinKwon + @yhuai Sorry it took so long! Things have been busy :)

HyukjinKwon · 2016-05-10T00:50:25Z

@RussellSpitzer Thanks for bearing with my pings here and there!

SparkQA · 2016-05-10T02:02:15Z

Test build #58192 has finished for PR 11317 at commit 117999f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

RussellSpitzer · 2016-05-10T03:46:33Z

I don't think this is because of me

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (sharedRuntime.cpp:834), pid=27637, tid=140000737998592
#  fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x00007f54dd188231
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/jenkins/workspace/SparkPullRequestBuilder@2/hs_err_pid27637.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Traceback (most recent call last):
  File "/usr/lib64/pypy-2.5.1/lib-python/2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib64/pypy-2.5.1/lib-python/2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib64/pypy-2.5.1/lib-python/2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/pypy-2.5.1/lib-python/2.7/SocketServer.py", line 655, in __init__
    self.handle()
  File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/accumulators.py", line 235, in handle
    num_updates = read_int(self.rfile)
  File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/serializers.py", line 545, in read_int
    raise EOFError
EOFError
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:40970)
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/lib/py4j-0.9.2-src.zip/py4j/java_gateway.py", line 713, in start
    self.socket.connect((self.address, self.port))
  File "<string>", line 1, in connect
error: [Errno 111] Connection refused

jacek-lewandowski · 2016-05-10T20:51:37Z

test this please

jacek-lewandowski · 2016-05-10T20:51:52Z

jenkins, test this please

SparkQA · 2016-05-10T22:24:42Z

Test build #58268 has finished for PR 11317 at commit 117999f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2016-07-06T04:03:50Z

@RussellSpitzer Sorry. I missed the last update on update. Would you please update the PR? I will review it and get it merged when it pass all tests.

RussellSpitzer · 2016-07-08T23:17:18Z

Updated

SparkQA · 2016-07-09T23:19:09Z

Test build #3176 has finished for PR 11317 at commit 4c23cf1.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

In order to make it clear which filters are fully handled by the underlying datasource we will mark them with a *. This will give a clear visual queue to users that the filter is being treated differently by catalyst than filters which are just presented to the underlying DataSource.

yhuai · 2016-07-11T05:25:00Z

tes this please

yhuai · 2016-07-11T22:23:02Z

tes thsi please

yhuai · 2016-07-11T22:23:09Z

ok to test

SparkQA · 2016-07-12T00:26:03Z

Test build #62122 has finished for PR 11317 at commit f0051c4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2016-07-12T04:39:01Z

lgtm. Merging to master.

yhuai reviewed Feb 23, 2016
View reviewed changes

asfgit closed this in b1e5281 Jul 12, 2016

RussellSpitzer deleted the SPARK-12639-Star branch September 15, 2016 15:24

RussellSpitzer mentioned this pull request Sep 15, 2016

[SPARK-12639][SQL] Improve Explain for Datasources with Handled Predicates #10655

Closed

[SPARK-12639] [SQL] Mark Filters Fully Handled By Sources with * #11317

[SPARK-12639] [SQL] Mark Filters Fully Handled By Sources with * #11317

Uh oh!

Conversation

RussellSpitzer commented Feb 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Before

After

How was the this patch tested?

Uh oh!

rxin commented Feb 23, 2016

Uh oh!

yhuai commented Feb 23, 2016

Uh oh!

yhuai commented Feb 23, 2016

Uh oh!

yhuai Feb 23, 2016

Choose a reason for hiding this comment

Uh oh!

yhuai commented Feb 23, 2016

Uh oh!

SparkQA commented Feb 23, 2016

Uh oh!

yhuai commented Mar 16, 2016

Uh oh!

SparkQA commented Mar 16, 2016

Uh oh!

HyukjinKwon commented May 7, 2016

Uh oh!

RussellSpitzer commented May 10, 2016

Uh oh!

HyukjinKwon commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

RussellSpitzer commented May 10, 2016

Uh oh!

jacek-lewandowski commented May 10, 2016

Uh oh!

jacek-lewandowski commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

yhuai commented Jul 6, 2016

Uh oh!

RussellSpitzer commented Jul 8, 2016

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

yhuai commented Jul 11, 2016

Uh oh!

yhuai commented Jul 11, 2016

Uh oh!

yhuai commented Jul 11, 2016

Uh oh!

SparkQA commented Jul 12, 2016

Uh oh!

yhuai commented Jul 12, 2016

Uh oh!

Uh oh!

RussellSpitzer commented Feb 23, 2016 •

edited

Loading