[SPARK-6953] [PySpark] speed up python tests

[davies]

This PR try to speed up some python tests:

tests.py                        144s -> 103s      -41s
mllib/classification.py   24s -> 17s          -7s
mllib/regression.py       27s -> 15s          -12s
mllib/tree.py                 27s  -> 13s         -14s
mllib/tests.py                64s -> 31s         -33s
streaming/tests.py       185s -> 84s        -101s

Considering python3, the total saving will be 558s (almost 10 minutes) (core, and streaming run three times, mllib runs twice).

During testing, it will show used time for each test file:

Run core tests ...
Running test: pyspark/rdd.py ... ok (22s)
Running test: pyspark/context.py ... ok (16s)
Running test: pyspark/conf.py ... ok (4s)
Running test: pyspark/broadcast.py ... ok (4s)
Running test: pyspark/accumulators.py ... ok (4s)
Running test: pyspark/serializers.py ... ok (6s)
Running test: pyspark/profiler.py ... ok (5s)
Running test: pyspark/shuffle.py ... ok (1s)
Running test: pyspark/tests.py ... ok (103s)   144s

[SparkQA]

Test build #29883 has finished for PR 5427 at commit be23e1d.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #29893 has finished for PR 5427 at commit 0c49785.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30386 has finished for PR 5427 at commit 945a2b5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30385 has finished for PR 5427 at commit fec2da2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UnresolvedAttribute(nameParts: Seq[String])
  • trait CaseConversionExpression
  • final class UTF8String extends Ordered[UTF8String] with Serializable
  • case class Exchange(
  • case class SortMergeJoin(
• This patch does not change any dependencies.

[mengxr]

@davies Did you measure the speedup on each component?

[davies]

@mengxr I had update the difference in the description.

[SparkQA]

Test build #30427 has finished for PR 5427 at commit 55bb451.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30444 has finished for PR 5427 at commit aa39c55.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30458 has finished for PR 5427 at commit 2654bfd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30516 has finished for PR 5427 at commit 2654bfd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[mengxr]

MLlib changes look good to me.

[rxin]

I brought this up to date at #5605