[SPARK-8492] [SQL] support binaryType in UnsafeRow

[davies]

Support BinaryType in UnsafeRow, just like StringType.

Also change the layout of StringType and BinaryType in UnsafeRow, by combining offset and size together as Long, which will limit the size of Row to under 2G (given that fact that any single buffer can not be bigger than 2G in JVM).

[davies]

cc @JoshRosen

[SparkQA]

Test build #35339 has finished for PR 6911 at commit 447dea0.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35346 has finished for PR 6911 at commit 6abfe93.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

The 2GB row limit isn't an issue since we already implicitly have that limit in UnsafeRowConverter.

[SparkQA]

Test build #35357 has finished for PR 6911 at commit 22e4c0a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35359 has finished for PR 6911 at commit 180b49d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #943 has finished for PR 6911 at commit 180b49d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • // class ParentClass(parentField: Int)
  • // class ChildClass(childField: Int) extends ParentClass(1)
  • // If the class type corresponding to current slot has writeObject() defined,
  • // then its not obvious which fields of the class will be serialized as the writeObject()
  • class SerializableConfiguration(@transient var value: Configuration) extends Serializable
  • class SerializableJobConf(@transient var value: JobConf) extends Serializable
  • class StreamingKMeansModel(KMeansModel):
  • class StreamingKMeans(object):
  • abstract class GeneratedClass
  • case class Bin(child: Expression)
  • case class Md5(child: Expression)

[SparkQA]

Test build #947 has finished for PR 6911 at commit 180b49d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

I guess the precedence is kind of obvious from usage / context, but it wouldn't hurt to add parens to disambiguate the order in which the shifts are applied.

[JoshRosen]

This looks good to me overall. I like the idea of storing the length in the fixed-length values section alongside the pointer to the variable-length data. I wonder whether there's a natural point to document / explicitly call out this encoding, though, in order to make it a bit more obvious to any new readers of this file.

[JoshRosen]

Do we need to mask out the upper 32 bits before converting to a long? I guess the uppermost bit probably can't be 1 because the offset can't be negative, so I guess we don't need to worry about sign-extension during the shift.

[davies]

No.

[SparkQA]

Test build #35479 has finished for PR 6911 at commit 519f698.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UnresolvedAlias(child: Expression) extends NamedExpression
  • abstract class ExtractValueWithStruct extends ExtractValue

[SparkQA]

Test build #35480 has finished for PR 6911 at commit d68706f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UnresolvedAlias(child: Expression) extends NamedExpression
  • abstract class ExtractValueWithStruct extends ExtractValue

[davies]

Merged into master