Skip to content

Commit e72a6ea

Browse files
author
chesterxgchen
committed
The Issue is due to that yarn-alpha and yarn have different APIs for certain class fields. In this particular case, the ClientBase using reflection to to address this issue, and we need to different way to test the ClientBase's method. Original ClientBaseSuite using getFieldValue() method to do this. But it doesn't work for yarn-alpha as the API returns an array of String instead of just String (which is the case for Yarn-stable API).
To fix the test, I add a new method def getFieldValue2[A: ClassTag, A1: ClassTag, B]( clazz: Class[_], field: String, defaults: => B) (mapTo: A => B)(mapTo1: A1 => B) : B = Try(clazz.getField(field)).map(_.get(null)).map { case v: A => mapTo(v) case v1: A1 => mapTo1(v1) case _ => defaults }.toOption.getOrElse(defaults) to handle the cases where the field type can be either type A or A1. In this new method the type A or A1 is pattern matched and corresponding mapTo function (mapTo or mapTo1) is used. Reformat based on Andrew's request
1 parent 4e3fbe8 commit e72a6ea

File tree

1 file changed

+16
-3
lines changed

1 file changed

+16
-3
lines changed

yarn/common/src/test/scala/org/apache/spark/deploy/yarn/ClientBaseSuite.scala

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ import org.scalatest.Matchers
3838

3939
import scala.collection.JavaConversions._
4040
import scala.collection.mutable.{ HashMap => MutableHashMap }
41+
import scala.reflect.ClassTag
4142
import scala.util.Try
4243

4344
import org.apache.spark.{SparkException, SparkConf}
@@ -200,9 +201,10 @@ class ClientBaseSuite extends FunSuite with Matchers {
200201

201202

202203
val knownDefMRAppCP: Seq[String] =
203-
getFieldValue[String, Seq[String]](classOf[MRJobConfig],
204-
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
205-
Seq[String]())(a => a.split(","))
204+
getFieldValue2[String, Array[String], Seq[String]](
205+
classOf[MRJobConfig],
206+
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
207+
Seq[String]())(a => a.split(","))(a => a.toSeq)
206208

207209
val knownYARNAppCP = Some(Seq("/known/yarn/path"))
208210

@@ -232,6 +234,17 @@ class ClientBaseSuite extends FunSuite with Matchers {
232234
def getFieldValue[A, B](clazz: Class[_], field: String, defaults: => B)(mapTo: A => B): B =
233235
Try(clazz.getField(field)).map(_.get(null).asInstanceOf[A]).toOption.map(mapTo).getOrElse(defaults)
234236

237+
def getFieldValue2[A: ClassTag, A1: ClassTag, B](
238+
clazz: Class[_],
239+
field: String,
240+
defaults: => B)(mapTo: A => B)(mapTo1: A1 => B) : B = {
241+
Try(clazz.getField(field)).map(_.get(null)).map {
242+
case v: A => mapTo(v)
243+
case v1: A1 => mapTo1(v1)
244+
case _ => defaults
245+
}.toOption.getOrElse(defaults)
246+
}
247+
235248
private class DummyClient(
236249
val args: ClientArguments,
237250
val conf: Configuration,

0 commit comments

Comments
 (0)