[SPARK-9167][SQL] use UTC Calendar in stringToDate

[cloud-fan]

fix 2 bugs introduced in #7353

1. we should use UTC Calendar when cast string to date . Before [SPARK-8995][SQL] cast date strings like '2015-01-01 12:15:31' to date #7353 , we use DateTimeUtils.fromJavaDate(Date.valueOf(s.toString)) to cast string to date, and fromJavaDate will call millisToDays to avoid the time zone issue. Now we use DateTimeUtils.stringToDate(s), we should create a Calendar with UTC in the begging.
2. we should not change the default time zone in test cases. The threadLocalLocalTimeZone and threadLocalTimestampFormat in DateTimeUtils will only be evaluated once for each thread, so we can't set the default time zone back anymore.

[cloud-fan]

cc @tarekauel @davies

[davies]

Instead of calling millisToDays, we could create a Calendar with UTC in the begging to avoid the timezone issues. Forget to comment on this when reviewing.

[SparkQA]

Test build #37704 has finished for PR 7488 at commit 21ef293.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class NaiveBayes(override val uid: String)
  • class KMeans(override val uid: String) extends Estimator[KMeansModel] with KMeansParams
  • class IntArrayParam(parent: Params, name: String, doc: String, isValid: Array[Int] => Boolean)
  • class KMeansModel(JavaModel):
  • class KMeans(JavaEstimator, HasFeaturesCol, HasMaxIter, HasSeed):
  • class PCA(JavaEstimator, HasInputCol, HasOutputCol):
  • class PCAModel(JavaModel):
  • abstract class Expression extends TreeNode[Expression] with Product
  • trait Generator extends Expression
  • abstract class UnaryLogExpression(f: Double => Double, name: String)
  • case class Conv(numExpr: Expression, fromBaseExpr: Expression, toBaseExpr: Expression)
  • case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
  • case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
  • case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")
  • trait NamedExpression extends Expression
  • abstract class Attribute extends LeafExpression with NamedExpression
  • case class IsNaN(child: Expression) extends UnaryExpression
  • abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging with Product
  • abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Product with Serializable
  • trait HashSemiJoin

[SparkQA]

Test build #37707 has finished for PR 7488 at commit 9cd6005.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please.

[SparkQA]

Test build #37711 has finished for PR 7488 at commit 9cd6005.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #34 has finished for PR 7488 at commit 9cd6005.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class NaiveBayes(override val uid: String)
  • class KMeans(override val uid: String) extends Estimator[KMeansModel] with KMeansParams
  • class IntArrayParam(parent: Params, name: String, doc: String, isValid: Array[Int] => Boolean)
  • logDebug("isMulticlass = " + metadata.isMulticlass)
  • * (i.e., if isMulticlass && isSpaceSufficientForAllCategoricalSplits),
  • logDebug("isMulticlass = " + metadata.isMulticlass)
  • class KMeansModel(JavaModel):
  • class KMeans(JavaEstimator, HasFeaturesCol, HasMaxIter, HasSeed):
  • class PCA(JavaEstimator, HasInputCol, HasOutputCol):
  • class PCAModel(JavaModel):
  • abstract class Expression extends TreeNode[Expression] with Product
  • abstract class UnsafeProjection extends Projection
  • case class FromUnsafeProjection(fields: Seq[DataType]) extends Projection
  • abstract class BaseProjection extends Projection
  • class SpecificProjection extends $
  • class SpecificProjection extends $
  • trait Generator extends Expression
  • abstract class UnaryLogExpression(f: Double => Double, name: String)
  • case class Conv(numExpr: Expression, fromBaseExpr: Expression, toBaseExpr: Expression)
  • case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
  • case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
  • case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")
  • trait NamedExpression extends Expression
  • abstract class Attribute extends LeafExpression with NamedExpression
  • case class IsNaN(child: Expression) extends UnaryExpression
  • abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging with Product
  • abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Product with Serializable
  • trait HashSemiJoin

[yjshen]

LGTM, tried this locally and fix the CastSuite test failure due to #7353 , hope to be merged ASAP.

[rxin]

Thanks - I've merged this.