[SPARK-2103][Streaming] Change to ClassTag for KafkaInputDStream and fix reflection issue #1508

jerryshao · 2014-07-21T06:40:02Z

This PR updates previous Manifest for KafkaInputDStream's Decoder to ClassTag, also fix the problem addressed in SPARK-2103.

Previous Java interface cannot actually get the type of Decoder, so when using this Manifest to reconstruct the decode object will meet reflection exception.

Also for other two Java interfaces, ClassTag[String] is useless because calling Scala API will get the right implicit ClassTag.

Current Kafka unit test cannot actually verify the interface. I've tested these interfaces in my local and distribute settings.

SparkQA · 2014-07-21T06:43:48Z

QA tests have started for PR 1508. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16900/consoleFull

srowen · 2014-07-21T07:29:08Z

Nice one Jerry! this actually enables using Kafka with non-String data in Java.

SparkQA · 2014-07-21T08:30:51Z

QA results for PR 1508:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16900/consoleFull

srowen · 2014-07-21T10:29:38Z

It's the MIMA test that fails, since the method signature is changed. It's possible to keep and deprecate the existing method of course. Should we just do that, or OK to remove the method on the grounds that the API doesn't quite work?

jerryshao · 2014-07-22T01:16:10Z

Hi @srowen , thanks for your advice. I've also noticed that seems MIMA cannot allow me to change the method signature. But I think it is meaningful and will not change the user's code, only need to recompile.

@tdas, would you mind giving me some suggestions, thanks a lot.

tdas · 2014-07-30T20:56:24Z

Jenkins, test this please.

SparkQA · 2014-07-30T20:58:52Z

QA tests have started for PR 1508. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17484/consoleFull

tdas · 2014-07-30T21:00:32Z

LGTM. I am okay with the binary compatibility change, since it is really wrong that we were using manifest instead of classtags.

But I am not sure I really understand what is the reason behind the failure of the
val keyDecoder = manifest[U].runtimeClass.getConstructor(classOf[VerifiableProperties])
Why could the manifest not get the runtimeClass correctly?

tdas · 2014-07-30T21:05:12Z

Oh, please add the appropriate exclude in the Mima exclusions
https://github.com/apache/spark/blob/master/project/MimaExcludes.scala

See the error in the jenkins console output for the exact exception string that you need to add.

SparkQA · 2014-07-30T21:47:16Z

QA results for PR 1508:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17484/consoleFull

tdas · 2014-07-30T21:52:39Z

This failure is unrelated to this patch. Its a current known issue in Jenkins, will run the test again when this error gets cleared.

jerryshao · 2014-07-31T01:24:14Z

Hi TD, thanks for your review. I think the previous issue is that:

implicit val keyCmd: Manifest[U] = implicitly[Manifest[AnyRef]].asInstanceOf[Manifest[U]]

This line of code cannot actually get the correct runtime class, the runtime class of keyCmd is Object as SPARK-2103 mentioned.

Because we only got Object, so create through reflection will get exception is this line:

val keyDecoder = manifest[U].runtimeClass.getConstructor(classOf[VerifiableProperties])

The right way is like this:

implicit val keyCmd: Manifest[U] = Manifest.classType(keyDecodeClass)

Manifest can also do the same thing if we correctly feed it with class info. But I think it would be better to change Manifest to ClassTag to keep align with other code, so I changed to use ClassTag.

I will add this to Mima exclude.

Thanks a lot.

tdas · 2014-07-31T01:26:47Z

I get it now. Let me try to run the Jenkins once again, to figure out what needs to be added to Mima

tdas · 2014-07-31T01:42:55Z

Jenkins, test this please.

SparkQA · 2014-07-31T01:49:05Z

QA tests have started for PR 1508. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17533/consoleFull

SparkQA · 2014-07-31T02:47:32Z

QA results for PR 1508:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17533/consoleFull

tdas · 2014-07-31T08:24:07Z

Again, this is an unrelated failure. Running it again.

tdas · 2014-07-31T16:50:35Z

Jenkins, test this please.

SparkQA · 2014-07-31T16:54:01Z

QA tests have started for PR 1508. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17584/consoleFull

SparkQA · 2014-07-31T17:48:00Z

QA results for PR 1508:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17584/consoleFull

tdas · 2014-07-31T17:50:17Z

Can you add these to the Mima excludes.

ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.streaming.kafka.KafkaUtils.createStream")

ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.streaming.kafka.KafkaReceiver.this")

The last one should not be necessary as it is not a public class. Let me look into this. No harm in adding the exclusion though.

SparkQA · 2014-08-01T02:04:24Z

QA tests have started for PR 1508. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17627/consoleFull

tdas · 2014-08-01T02:07:19Z

@jerryshao Oops, not sure how that is possible given what this patch touches. Can you merge with master nonetheless.

jerryshao · 2014-08-01T02:10:51Z

Ok, I will update the code.

…ecoder construct issue when using Java API

SparkQA · 2014-08-01T02:39:23Z

QA tests have started for PR 1508. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17629/consoleFull

SparkQA · 2014-08-01T03:30:35Z

QA results for PR 1508:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17629/consoleFull

SparkQA · 2014-08-01T03:47:18Z

QA results for PR 1508:
- This patch PASSES unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17627/consoleFull

tdas · 2014-08-01T06:17:07Z

OK, this is very confusing, sequence of two results is very confusing. Let me run the tests again.

tdas · 2014-08-01T06:17:15Z

Jenkins, test this again.

tdas · 2014-08-01T10:26:31Z

Jenkins, test this please.

SparkQA · 2014-08-01T10:29:10Z

QA tests have started for PR 1508. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17670/consoleFull

srowen · 2014-08-01T10:29:19Z

A bunch of tests have been failing spuriously with "java.net.BindException: Address already in use". It's not the PR. I wonder what recent change could have made this happen? Is something being stricter about assigning a fixed port? did a config change to let multiple tests run on the same virtual machine?

tdas · 2014-08-01T10:30:26Z

Also, a number of them have been failing for spurious python mllib issues. At least was the case yesterday.

SparkQA · 2014-08-01T11:21:03Z

QA results for PR 1508:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17670/consoleFull

tdas · 2014-08-01T11:31:11Z

Heyyyaa, it finally passed. I am merging this. Thanks @jerryshao

…fix reflection issue This PR updates previous Manifest for KafkaInputDStream's Decoder to ClassTag, also fix the problem addressed in [SPARK-2103](https://issues.apache.org/jira/browse/SPARK-2103). Previous Java interface cannot actually get the type of Decoder, so when using this Manifest to reconstruct the decode object will meet reflection exception. Also for other two Java interfaces, ClassTag[String] is useless because calling Scala API will get the right implicit ClassTag. Current Kafka unit test cannot actually verify the interface. I've tested these interfaces in my local and distribute settings. Author: jerryshao <saisai.shao@intel.com> Closes apache#1508 from jerryshao/SPARK-2103 and squashes the following commits: e90c37b [jerryshao] Add Mima excludes 7529810 [jerryshao] Change Manifest to ClassTag for KafkaInputDStream's Decoder and fix Decoder construct issue when using Java API

salex89 · 2015-02-16T14:38:24Z

I'm not sure if this is the exactly same issue, but I am experience this with Spark 1.2.1 and Scala 2.10.4 also.

ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 -   java.lang.NoSuchMethodException: UserLocationEventDecoder.<init>(kafka.utils.VerifiableProperties)
at java.lang.Class.getConstructor0(Class.java:2971)
at java.lang.Class.getConstructor(Class.java:1812)
at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:106)
at   org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
at   org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:288)
at   org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:280)
at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1353)
at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1353)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

jerryshao · 2015-02-20T02:50:40Z

Hi @salex89 , seems your exception is a little different to this PR, maybe I need a careful look at this exception, thanks for your reporting.

salex89 · 2015-02-20T08:19:07Z

@jerryshao
OK, sorry for necroposting then. Should I open an issue on JIRA with the snippet?

jerryshao · 2015-02-21T06:01:22Z

Yeah, please do it and describe it in detail so we can easily reproduce it.

…locks.enabled` by default (apache#1508) ### What changes were proposed in this pull request? rdar://99068608 (SPARK-40198 Enable `spark.storage.decommission.(rdd|shuffle)Blocks.enabled` by default) This PR aims to enable the following two configurations by default. Note that these storage configurations are disabled by default by the parent configuration, `spark.storage.decommission.enabled`, still. In addition, all decommission features including this storage decommissioning are disabled by `spark.decommission.enabled` by default. - `spark.storage.decommission.rddBlocks.enabled` - `spark.storage.decommission.shuffleBlocks.enabled` ### Why are the changes needed? This will help users use the storage decommissioning feature more easily. ### Does this PR introduce _any_ user-facing change? Yes, this will reduce the number of required configurations for decommission users. However, these two configurations are disabled by two configurations still. ### How was this patch tested? Pass the CIs.

jerryshao mentioned this pull request Jul 31, 2014

[SPARK-1022] Kafka unit test that actually sends and receives data #557

Closed

jerryshao added 2 commits August 1, 2014 10:16

Change Manifest to ClassTag for KafkaInputDStream's Decoder and fix D…

7529810

…ecoder construct issue when using Java API

Add Mima excludes

e90c37b

asfgit closed this in a32f0fb Aug 1, 2014

[SPARK-2103][Streaming] Change to ClassTag for KafkaInputDStream and fix reflection issue #1508

[SPARK-2103][Streaming] Change to ClassTag for KafkaInputDStream and fix reflection issue #1508

Uh oh!

Conversation

jerryshao commented Jul 21, 2014

Uh oh!

SparkQA commented Jul 21, 2014

Uh oh!

srowen commented Jul 21, 2014

Uh oh!

SparkQA commented Jul 21, 2014

Uh oh!

srowen commented Jul 21, 2014

Uh oh!

jerryshao commented Jul 22, 2014

Uh oh!

tdas commented Jul 30, 2014

Uh oh!

SparkQA commented Jul 30, 2014

Uh oh!

tdas commented Jul 30, 2014

Uh oh!

tdas commented Jul 30, 2014

Uh oh!

SparkQA commented Jul 30, 2014

Uh oh!

tdas commented Jul 30, 2014

Uh oh!

jerryshao commented Jul 31, 2014

Uh oh!

tdas commented Jul 31, 2014

Uh oh!

tdas commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

tdas commented Jul 31, 2014

Uh oh!

tdas commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

tdas commented Jul 31, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

tdas commented Aug 1, 2014

Uh oh!

jerryshao commented Aug 1, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

tdas commented Aug 1, 2014

Uh oh!

tdas commented Aug 1, 2014

Uh oh!

tdas commented Aug 1, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

srowen commented Aug 1, 2014

Uh oh!

tdas commented Aug 1, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

tdas commented Aug 1, 2014

Uh oh!

salex89 commented Feb 16, 2015

Uh oh!

jerryshao commented Feb 20, 2015

Uh oh!