[SPARK-14796][SQL] Add spark.sql.optimizer.inSetConversionThreshold config option.

[dongjoon-hyun]

What changes were proposed in this pull request?

Currently, OptimizeIn optimizer replaces In expression into InSet expression if the size of set is greater than a constant, 10.
This issue aims to make a configuration spark.sql.optimizer.inSetConversionThreshold for that.

After this PR, OptimizerIn is configurable.

scala> sql("select a in (1,2,3) from (select explode(array(1,2)) a) T").explain()
== Physical Plan ==
WholeStageCodegen
:  +- Project [a#7 IN (1,2,3) AS (a IN (1, 2, 3))#8]
:     +- INPUT
+- Generate explode([1,2]), false, false, [a#7]
   +- Scan OneRowRelation[]

scala> sqlContext.setConf("spark.sql.optimizer.inSetConversionThreshold", "2")

scala> sql("select a in (1,2,3) from (select explode(array(1,2)) a) T").explain()
== Physical Plan ==
WholeStageCodegen
:  +- Project [a#16 INSET (1,2,3) AS (a IN (1, 2, 3))#17]
:     +- INPUT
+- Generate explode([1,2]), false, false, [a#16]
   +- Scan OneRowRelation[]

How was this patch tested?

Pass the Jenkins tests (with a new testcase)

[rxin]

Can we add a unit test in the appropriate optimizer suite?

We also need to come up with a better name. optimizer.minSetSize is not very intuitive.

[dongjoon-hyun]

Thank you for review, @rxin .
I'll add a OptimizeInSuite.scala for this.
For the name, could you give some advice?

[SparkQA]

Test build #56492 has finished for PR 12562 at commit 1ac5857.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class OptimizeIn(conf: CatalystConf) extends Rule[LogicalPlan]

[dongjoon-hyun]

Oh, sorry. There exists already OptimizeInSuite.scala. I'll fix soon.

[SparkQA]

Test build #56511 has finished for PR 12562 at commit 69306cc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #56514 has finished for PR 12562 at commit 75e1577.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #56550 has finished for PR 12562 at commit 9df35bb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class OptimizeIn(conf: CatalystConf) extends Rule[LogicalPlan]

[rxin]

I'd give this a more descriptive name, and explicitly say setting the threshold for turning into InSet

[rxin]

maybe

inSetConversionThreshold?

[marmbrus]

Do you use this import?

[dongjoon-hyun]

Thank you so much, @rxin and @marmbrus !
I will update soon like the following according to the comments.

• spark.sql.optimizer.minSetSize -> spark.sql.optimizer.inSetConversionThreshold
• SQLConf.optimizerMinSetSize, CatalystConf.optimizerMinSetSize -> optimizerInSetConversionThreshold
• SQLConf.OPTIMIZER_MIN_SET_SIZE -> SQLConf.OPTIMIZER_INSET_CONVERSION_THRESHOLD (also the description)
• and other comments.

By the way, @marmbrus d you mean the duplication of value 10 in SQLConf.scala and SimpleCatalystConf?

[SparkQA]

Test build #56573 has finished for PR 12562 at commit c7a6d9b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @rxin and @marmbrus .
How do you think about the updated PR? It's just second update.
If there is something to do more, please let me know.
Thank you.

[rxin]

Merging in master. Thanks.

[dongjoon-hyun]

Thank you, @rxin and @marmbrus .