[SPARK-3665][GraphX] Java API for GraphX

[ankurdave]

Adds JavaGraph (which also contains methods from GraphOps), JavaVertexRDD, JavaEdgeRDD, and tests for these classes.

[SparkQA]

Test build #23285 has finished for PR 3234 at commit 43cf4a3.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class JavaSimpleTextClassificationPipeline
  • case class LabeledDocument(id: Long, text: String, label: Double)
  • case class Document(id: Long, text: String)
  • abstract class EdgeRDD[ED, VD](
  • abstract class VertexRDD[VD](
  • class JavaEdgeRDD[ED, VD](override val rdd: EdgeRDD[ED, VD])(
  • class JavaGraph[VD, ED](val graph: Graph[VD, ED])(
  • class JavaVertexRDD[VD](val vertexRDD: VertexRDD[VD])(implicit val vdTag: ClassTag[VD])
  • public class PartitionStrategies
  • abstract class Estimator[M <: Model[M]] extends PipelineStage with Params
  • abstract class Evaluator extends Identifiable
  • abstract class Model[M <: Model[M]] extends Transformer
  • abstract class PipelineStage extends Serializable with Logging
  • class Pipeline extends Estimator[PipelineModel]
  • abstract class Transformer extends PipelineStage with Params
  • class LogisticRegression extends Estimator[LogisticRegressionModel] with LogisticRegressionParams
  • class HashingTF extends UnaryTransformer[Iterable[_], Vector, HashingTF]
  • class StandardScaler extends Estimator[StandardScalerModel] with StandardScalerParams
  • class Tokenizer extends UnaryTransformer[String, Seq[String], Tokenizer]
  • class Param[T] (
  • class DoubleParam(parent: Params, name: String, doc: String, defaultValue: Option[Double] = None)
  • class IntParam(parent: Params, name: String, doc: String, defaultValue: Option[Int] = None)
  • class FloatParam(parent: Params, name: String, doc: String, defaultValue: Option[Float] = None)
  • class LongParam(parent: Params, name: String, doc: String, defaultValue: Option[Long] = None)
  • class BooleanParam(parent: Params, name: String, doc: String, defaultValue: Option[Boolean] = None)
  • case class ParamPair[T](param: Param[T], value: T)
  • trait Params extends Identifiable with Serializable
  • class CrossValidator extends Estimator[CrossValidatorModel] with CrossValidatorParams with Logging
  • class ParamGridBuilder

[ankurdave]

Git fetch failure. Jenkins, retest this please

[SparkQA]

Test build #23289 has finished for PR 3234 at commit b2d6590.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class JavaSimpleTextClassificationPipeline
  • case class LabeledDocument(id: Long, text: String, label: Double)
  • case class Document(id: Long, text: String)
  • abstract class EdgeRDD[ED, VD](
  • abstract class VertexRDD[VD](
  • class JavaEdgeRDD[ED, VD](override val rdd: EdgeRDD[ED, VD])(
  • class JavaGraph[VD, ED](val graph: Graph[VD, ED])(
  • class JavaVertexRDD[VD](val vertexRDD: VertexRDD[VD])(implicit val vdTag: ClassTag[VD])
  • public class PartitionStrategies
  • abstract class Estimator[M <: Model[M]] extends PipelineStage with Params
  • abstract class Evaluator extends Identifiable
  • abstract class Model[M <: Model[M]] extends Transformer
  • abstract class PipelineStage extends Serializable with Logging
  • class Pipeline extends Estimator[PipelineModel]
  • abstract class Transformer extends PipelineStage with Params
  • class LogisticRegression extends Estimator[LogisticRegressionModel] with LogisticRegressionParams
  • class HashingTF extends UnaryTransformer[Iterable[_], Vector, HashingTF]
  • class StandardScaler extends Estimator[StandardScalerModel] with StandardScalerParams
  • class Tokenizer extends UnaryTransformer[String, Seq[String], Tokenizer]
  • class Param[T] (
  • class DoubleParam(parent: Params, name: String, doc: String, defaultValue: Option[Double] = None)
  • class IntParam(parent: Params, name: String, doc: String, defaultValue: Option[Int] = None)
  • class FloatParam(parent: Params, name: String, doc: String, defaultValue: Option[Float] = None)
  • class LongParam(parent: Params, name: String, doc: String, defaultValue: Option[Long] = None)
  • class BooleanParam(parent: Params, name: String, doc: String, defaultValue: Option[Boolean] = None)
  • case class ParamPair[T](param: Param[T], value: T)
  • trait Params extends Identifiable with Serializable
  • class CrossValidator extends Estimator[CrossValidatorModel] with CrossValidatorParams with Logging
  • class ParamGridBuilder

[SparkQA]

Test build #23288 has finished for PR 3234 at commit b2d6590.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class JavaSimpleTextClassificationPipeline
  • case class LabeledDocument(id: Long, text: String, label: Double)
  • case class Document(id: Long, text: String)
  • abstract class EdgeRDD[ED, VD](
  • abstract class VertexRDD[VD](
  • class JavaEdgeRDD[ED, VD](override val rdd: EdgeRDD[ED, VD])(
  • class JavaGraph[VD, ED](val graph: Graph[VD, ED])(
  • class JavaVertexRDD[VD](val vertexRDD: VertexRDD[VD])(implicit val vdTag: ClassTag[VD])
  • public class PartitionStrategies
  • abstract class Estimator[M <: Model[M]] extends PipelineStage with Params
  • abstract class Evaluator extends Identifiable
  • abstract class Model[M <: Model[M]] extends Transformer
  • abstract class PipelineStage extends Serializable with Logging
  • class Pipeline extends Estimator[PipelineModel]
  • abstract class Transformer extends PipelineStage with Params
  • class LogisticRegression extends Estimator[LogisticRegressionModel] with LogisticRegressionParams
  • class HashingTF extends UnaryTransformer[Iterable[_], Vector, HashingTF]
  • class StandardScaler extends Estimator[StandardScalerModel] with StandardScalerParams
  • class Tokenizer extends UnaryTransformer[String, Seq[String], Tokenizer]
  • class Param[T] (
  • class DoubleParam(parent: Params, name: String, doc: String, defaultValue: Option[Double] = None)
  • class IntParam(parent: Params, name: String, doc: String, defaultValue: Option[Int] = None)
  • class FloatParam(parent: Params, name: String, doc: String, defaultValue: Option[Float] = None)
  • class LongParam(parent: Params, name: String, doc: String, defaultValue: Option[Long] = None)
  • class BooleanParam(parent: Params, name: String, doc: String, defaultValue: Option[Boolean] = None)
  • case class ParamPair[T](param: Param[T], value: T)
  • trait Params extends Identifiable with Serializable
  • class CrossValidator extends Estimator[CrossValidatorModel] with CrossValidatorParams with Logging
  • class ParamGridBuilder

[rxin]

Ok since this is so big, we should probably put this for 1.3 now :)

[andrewor14]

Hi @ankurdave is this still targeted for 1.4 now?

[brennonyork]

Would also like to know the status. This looks to be the last piece to graduate GraphX from alpha to stable re: SPARK-3664.

[rxin]

@brennonyork graphx has already graduated from alpha. We will make sure this goes into 1.4.

[sryza]

Mind tagging this with [GRAPHX] so it can get sorted properly?

[kdatta]

I had to add the Junit dependency in graphx/pom.xml to compile. Did you see this issue? We might have to update the pom file.

-Kushal.

[srowen]

@ankurdave what happened to this? should this PR be closed?

[JoshRosen]

Hey @ankurdave, it looks like this PR has gone stale, so do you mind closing it for now and re-opening it once you're ready to work on it again?

[kdatta]

Hi Josh, Ankur,

We have a dependence on this story for PyGraphX. We are close to completing the PyGraphX work and would like this code to be reviewed and approved.

Thanks,
-Kushal.

From: Josh Rosen [mailto:notifications@github.com]
Sent: Monday, June 29, 2015 8:51 AM
To: apache/spark
Cc: Datta, Kushal
Subject: Re: [spark] [SPARK-3665][GraphX] Java API for GraphX (#3234)

Hey @ankurdavehttps://github.com/ankurdave, it looks like this PR has gone stale, so do you mind closing it for now and re-opening it once you're ready to work on it again?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3234#issuecomment-116739058.

[srowen]

(For Jenkins:) Do you mind closing this PR then?

[kdatta]

Don't mind at all. We will submit the design doc of PyGraphX by early next week. Will close this PR after.

[ankurdave]

@JoshRosen @srowen Sorry I didn't see your requests to close earlier. Closing now.