Skip to content

Commit fa75db2

Browse files
[SPARK-29026][SQL] Improve error message in schemaFor in trait without companion object constructor
### What changes were proposed in this pull request? - For trait without companion object constructor, currently the method to get constructor parameters `constructParams` in `ScalaReflection` will throw exception. ``` scala.ScalaReflectionException: <none> is not a term at scala.reflect.api.Symbols$SymbolApi.asTerm(Symbols.scala:211) at scala.reflect.api.Symbols$SymbolApi.asTerm$(Symbols.scala:211) at scala.reflect.internal.Symbols$SymbolContextApiImpl.asTerm(Symbols.scala:106) at org.apache.spark.sql.catalyst.ScalaReflection.getCompanionConstructor(ScalaReflection.scala:909) at org.apache.spark.sql.catalyst.ScalaReflection.constructParams(ScalaReflection.scala:914) at org.apache.spark.sql.catalyst.ScalaReflection.constructParams$(ScalaReflection.scala:912) at org.apache.spark.sql.catalyst.ScalaReflection$.constructParams(ScalaReflection.scala:47) at org.apache.spark.sql.catalyst.ScalaReflection.getConstructorParameters(ScalaReflection.scala:890) at org.apache.spark.sql.catalyst.ScalaReflection.getConstructorParameters$(ScalaReflection.scala:886) at org.apache.spark.sql.catalyst.ScalaReflection$.getConstructorParameters(ScalaReflection.scala:47) ``` - Instead this PR would throw exception: ``` Unable to find constructor for type [XXX]. This could happen if [XXX] is an interface or a trait without companion object constructor UnsupportedOperationException: ``` In the normal usage of ExpressionEncoder, this can happen if the type is interface extending `scala.Product`. Also, since this is a protected method, this could have been other arbitrary types without constructor. ### Why are the changes needed? - The error message `<none> is not a term` isn't helpful for users to understand the problem. ### Does this PR introduce any user-facing change? - The exception would be thrown instead of runtime exception from the `scala.ScalaReflectionException`. ### How was this patch tested? - Added a unit test to illustrate the `type` where expression encoder will fail and trigger the proposed error message. Closes #25736 from mickjermsurawong-stripe/SPARK-29026. Authored-by: Mick Jermsurawong <mickjermsurawong@stripe.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
1 parent 54d3f6e commit fa75db2

File tree

2 files changed

+34
-1
lines changed

2 files changed

+34
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -906,7 +906,18 @@ trait ScalaReflection extends Logging {
906906
* only defines a constructor via `apply` method.
907907
*/
908908
private def getCompanionConstructor(tpe: Type): Symbol = {
909-
tpe.typeSymbol.asClass.companion.asTerm.typeSignature.member(universe.TermName("apply"))
909+
def throwUnsupportedOperation = {
910+
throw new UnsupportedOperationException(s"Unable to find constructor for $tpe. " +
911+
s"This could happen if $tpe is an interface, or a trait without companion object " +
912+
"constructor.")
913+
}
914+
tpe.typeSymbol.asClass.companion match {
915+
case NoSymbol => throwUnsupportedOperation
916+
case sym => sym.asTerm.typeSignature.member(universe.TermName("apply")) match {
917+
case NoSymbol => throwUnsupportedOperation
918+
case constructorSym => constructorSym
919+
}
920+
}
910921
}
911922

912923
protected def constructParams(tpe: Type): Seq[Symbol] = {

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,14 @@ trait ScroogeLikeExample extends Product1[Int] with Serializable {
138138
override def hashCode: Int = x
139139
}
140140

141+
/** Counter-examples to [[ScroogeLikeExample]] as a trait without a companion object constructor */
142+
trait TraitProductWithoutCompanion extends Product1[Int] {}
143+
144+
/** Counter-examples to [[ScroogeLikeExample]] as a trait with no-constructor companion object */
145+
object TraitProductWithNoConstructorCompanion {}
146+
147+
trait TraitProductWithNoConstructorCompanion extends Product1[Int] {}
148+
141149
class ScalaReflectionSuite extends SparkFunSuite {
142150
import org.apache.spark.sql.catalyst.ScalaReflection._
143151

@@ -404,6 +412,20 @@ class ScalaReflectionSuite extends SparkFunSuite {
404412
StructField("x", IntegerType, nullable = false))), nullable = true))
405413
}
406414

415+
test("SPARK-29026: schemaFor for trait without companion object throws exception ") {
416+
val e = intercept[UnsupportedOperationException] {
417+
schemaFor[TraitProductWithoutCompanion]
418+
}
419+
assert(e.getMessage.contains("Unable to find constructor"))
420+
}
421+
422+
test("SPARK-29026: schemaFor for trait with no-constructor companion throws exception ") {
423+
val e = intercept[UnsupportedOperationException] {
424+
schemaFor[TraitProductWithNoConstructorCompanion]
425+
}
426+
assert(e.getMessage.contains("Unable to find constructor"))
427+
}
428+
407429
test("SPARK-27625: annotated data types") {
408430
assert(serializerFor[FooWithAnnotation].dataType == StructType(Seq(
409431
StructField("f1", StringType),

0 commit comments

Comments
 (0)