[SPARK-8753][SQL] Create an IntervalType data type

[cloud-fan]

We need a new data type to represent time intervals. Because we can't determine how many days in a month, so we need 2 values for interval: a int months, a long microseconds.

The interval literal syntax looks like:
interval 3 years -4 month 4 weeks 3 second

Because we use number of 100ns as value of TimestampType, so it may not makes sense to support nano second unit.

[cloud-fan]

cc @rxin

[SparkQA]

Test build #36544 has finished for PR 7226 at commit 0502af9.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36547 has finished for PR 7226 at commit 1adbae1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

let's create an abstract data type for IntervalType.

[rxin]

We should add a way to cast interval into string and string to interval too. That can go in a separate pull request though.

[SparkQA]

Test build #36572 has finished for PR 7226 at commit 9129ab0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

@cloud-fan actually upon further thought, I think it makes more sense to have a single IntervalType and internally represent it as a 12-byte value (1 long + 1 int). We can create a new class for it.

[SparkQA]

Test build #36683 has finished for PR 7226 at commit 3c641a0.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36684 has finished for PR 7226 at commit 296dfb6.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36685 has finished for PR 7226 at commit 504456c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36691 has finished for PR 7226 at commit b1fc20f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Interval(months: Int, microseconds: Long)

[rxin]

this is not a legit comparison actually -- i'm not sure if you can sort directly on interval data types this way.

(the problem is that the number of seconds or days can be greater than month)

[cloud-fan]

Can we just assume there are 30 days in a month and transform months to microseconds and compare them?

[cloud-fan]

I didn't make it extend AtomicType here, as I haven't figured out how to compare intervals. 30 days and 1 months may have different compare result in different context.

[SparkQA]

Test build #36757 has finished for PR 7226 at commit ac348c3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Interval(months: Int, microseconds: Long) extends Serializable

[rxin]

when is this called? if it is called during analysis, it'd make more sense to throw AnalysisException, since that has better error reporting in Python.

[rxin]

looks good otherwise.

[SparkQA]

Test build #36787 has finished for PR 7226 at commit 43ccc80.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36789 has finished for PR 7226 at commit 632062d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public final class Interval implements Serializable

[rxin]

Thanks - merging this in.

[rxin]

can you implement to string here?

[rxin]

I filed https://issues.apache.org/jira/browse/SPARK-8938 for it.

[liancheng]

Should we make IntervalType an AtomicType?

[cloud-fan]

hi @liancheng , I didn't make it extend AtomicType, as I haven't figured out how to compare intervals. 30 days and 1 months may have different compare result in different context.

[liancheng]

@cloud-fan Thanks for the explanation :)