[SPARK-1022] Kafka unit test that actually sends and receives data

[tdas]

This was originally done by @jerryshao in the incubator repository. I cherry picked him commits and made a new PR here. However, the tests are flaky so they are ignored for now.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[tdas]

@jerryshao It would be great if you can take a look at this sometime. The tests seems to fail 1 in 3 times as the receiver does not receive any data. Also there are a number of warning and errors in the log when the test are run, some of which are particularly disturbing (method not found in ZkClient!)

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14491/

[jerryshao]

Hi TD, thanks for bringing it out, I will checkout your branch to see the details.

[jerryshao]

Hi TD, the exception (method not found in ZkClient) is introduced by zkclient-0.1.jar. Actually Kafka 0.8 relies on zkclient-0.3.jar which will be automatically imported by Kafka. So by removing the dependency line in build profile: "com.github.sgroschupf" % "zkclient" % "0.1" excludeAll(excludeNetty), can remove this exception.

As tests seems to fail unpredictably, it also confuses me a lot, I'm not sure it's the problem of producer do not produce any data, or the problem of receiver do not receive any data. I will trace it to see the details if I can.

[tdas]

@jerryshao I know this is old stuff but did you get a chance to take a look further into this? If not, it would be great if you can while working on other kafka + streaming stuff. :)

[jerryshao]

Yes, I did revisit this patch again long ago, but as you mentioned it is so flaky. I guess the main reason is that our Kafka API call is in high level which involves Zookeeper's behavior (unknown latency or others), so our unit test cannot precisely reproduce the result each time. I think if we shift to some low level API, we can have more power to control the behavior, this problem can be easily fixed.

[tdas]

Its weird that zookeeper behavior is so unpredictable that this test would
be so flaky. If we make the low-level API we should still keep the
high-level API for backward compatibility for a while, so it would be good
if a fix for this can be figured out.

TD

On Thu, Jul 10, 2014 at 6:07 PM, Saisai Shao notifications@github.com
wrote:

Yes, I did revisit this patch again long ago, but as you mentioned it is
so flaky. I guess the main reason is that our Kafka API call is in high
level which involves Zookeeper's behavior (unknown latency or others), so
our unit test cannot precisely reproduce the result each time. I think if
we shift to some low level API, we can have more power to control the
behavior, this problem can be easily fixed.

—
Reply to this email directly or view it on GitHub
#557 (comment).

[jerryshao]

I'm not super familiar with Kafka's internal, so this unit test is mostly taken Kafka's unit test as references, there might be some miss use without diving into the details deeply. I'll try to fix this, really depends on my knowledge of Kafka :)

[jerryshao]

Hi TD, I've fixed this flaky issue in my local test, how should I submit my patch, should I push to your branch or create a new PR?

[tdas]

Dang, I missed this update. Go ahead and make a new PR. I can close this PR.
Incidentally, what was the issue?

[jerryshao]

I think the main reason is that some times when we feed our test data to Kafka, the KafkaReceiver is still starting, after the receiver started, it will ignore the previous sent data and read from the end of the queue (because of the behavior Kafka 0.8's auto.offset.reset=largest), so previous sent test data will never be consumed, and the test will be failed. So the simple way is to sleep for several seconds after ssc.start() and then feed the data.

[jerryshao]

I will create another PR, thanks a lot. I think it would be better to merge (#1508) before, so we can also test the Java API correctly. Thanks a lot.

[tdas]

Yeah, will merge #1508 as soon as the tests pass.

[tdas]

@jerryshao Since #1508 has been merged, can you update this and test it to make sure this works?

[jerryshao]

Yeah, I will submit to new PR about this ASAP :).