[SPARK-3054][STREAMING] Add unit tests for Spark Sink. #1958

harishreedharan · 2014-08-15T00:27:44Z

This patch adds unit tests for Spark Sink.

It also removes the private[flume] for Spark Sink,
since the sink is instantiated from Flume configuration (looks like this is ignored by reflection which is used by
Flume, but we should still remove it anyway).

This patch adds unit tests for Spark Sink. It also removes the private[flume] for Spark Sink, since the sink is instantiated from Flume configuration (looks like this is ignored by reflection which is used by Flume, but we should still remove it anyway).

SparkQA · 2014-08-15T00:30:22Z

QA tests have started for PR 1958. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18583/consoleFull

SparkQA · 2014-08-15T01:22:48Z

QA results for PR 1958:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18583/consoleFull

jerryshao · 2014-08-15T05:06:06Z

external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala

+
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more


Hey Hari, ASF header should be at the top of file :).

Thanks! Done.

SparkQA · 2014-08-15T06:20:10Z

QA tests have started for PR 1958 at commit f2c56c9.

This patch merges cleanly.

harishreedharan · 2014-08-15T16:21:52Z

Jenkins, test this please

SparkQA · 2014-08-15T16:25:15Z

QA tests have started for PR 1958 at commit f2c56c9.

This patch merges cleanly.

SparkQA · 2014-08-15T17:11:09Z

QA tests have finished for PR 1958 at commit f2c56c9.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

harishreedharan · 2014-08-15T17:58:27Z

Failure is unrelated:
[info] SparkSinkSuite:
[info] - Success test
[info] - Nack
[info] - Timeout
[info] - Multiple consumers
[info] - Multiple consumers With Some Failures
[info] ScalaTest
[info] Run completed in 10 minutes, 40 seconds.
[info] Total number of tests run: 5
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0

Failures are in the SparkSubmitSuite:
[info] 2 TESTS FAILED
[error] Failed: Total 829, Failed 2, Errors 0, Passed 827, Ignored 6
[error] Failed tests:
[error] org.apache.spark.deploy.SparkSubmitSuite
error sbt.TestsFailedException: Tests unsuccessful

SparkQA · 2014-08-15T18:10:22Z

QA tests have started for PR 1958 at commit 7b9b649.

This patch merges cleanly.

SparkQA · 2014-08-15T18:58:18Z

QA tests have finished for PR 1958 at commit 7b9b649.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

harishreedharan · 2014-08-16T00:01:24Z

Same failures as before - not caused by this patch

harishreedharan · 2014-08-16T00:28:11Z

Jenkins, test this please

SparkQA · 2014-08-18T17:25:18Z

QA tests have started for PR 1958 at commit 7fedc5a.

This patch merges cleanly.

SparkQA · 2014-08-18T18:14:07Z

QA tests have finished for PR 1958 at commit 7fedc5a.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2014-08-18T21:45:08Z

external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSink.scala

@@ -53,7 +53,6 @@ import org.apache.flume.sink.AbstractSink
 *
 */

-private[flume]


Why was this removed? We dont want to expose this as a public class as this class will then appear in the Scala docs.

Since this class would be called from Flume. Flume will create an instance of this class to run the sink - so theoretically it should not be private to this package.

In that case, can you add a line right at the top that this class is not intended to be used inside Spark application. Just in case it appears in the scala docs / java docs. I will try to see how to eliminate this module from appearing in the docs.

SparkQA · 2014-08-18T22:10:42Z

QA tests have started for PR 1958 at commit c9190d1.

This patch merges cleanly.

SparkQA · 2014-08-18T22:53:55Z

QA tests have finished for PR 1958 at commit c9190d1.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2014-08-19T06:59:52Z

unit test is failing.

tdas · 2014-08-19T07:19:57Z

external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala

+    transAndClient.foreach(x => x._1.close())
+  }
+
+  def initializeChannelAndSink(overrides: Option[Map[String, String]]): (MemoryChannel,


nit: This is a little weird. Option is not necessary (add None and Some unnecessarily increases verbosity), as it can simply be def initializeChannelAndSink(overrides: Map[String, String] = Map.empty)

Also, add private to keep it consistent to other methods.

harishreedharan · 2014-08-19T22:27:29Z

Looks like the tests have passed.

tdas · 2014-08-19T22:28:55Z

Right, good. But I want to run it another couple of times to test its flakiness.

SparkQA · 2014-08-19T22:38:16Z

QA tests have finished for PR 1958 at commit e3110b9.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2014-08-19T22:41:07Z

Jenkins, test this.

harishreedharan · 2014-08-19T23:15:19Z

Jenkins, test this please

harishreedharan · 2014-08-20T03:44:34Z

Jenkins, test this please

SparkQA · 2014-08-20T03:50:23Z

QA tests have started for PR 1958 at commit e3110b9.

This patch merges cleanly.

SparkQA · 2014-08-20T03:50:30Z

QA tests have finished for PR 1958 at commit e3110b9.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

harishreedharan · 2014-08-20T03:52:40Z

Could not find Apache license headers in the following files:
!????? /home/jenkins/workspace/SparkPullRequestBuilder/mllib/checkpoint/.temp.crc
!????? /home/jenkins/workspace/SparkPullRequestBuilder/mllib/checkpoint/temp

No idea what is happening here.

tdas · 2014-08-20T06:06:11Z

Jenkins, test this please.

SparkQA · 2014-08-20T06:10:25Z

QA tests have started for PR 1958 at commit e3110b9.

This patch merges cleanly.

SparkQA · 2014-08-20T06:57:17Z

QA tests have finished for PR 1958 at commit e3110b9.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2014-08-20T07:05:33Z

Jenkins, test this please.

SparkQA · 2014-08-20T07:10:23Z

QA tests have started for PR 1958 at commit e3110b9.

This patch merges cleanly.

SparkQA · 2014-08-20T08:04:49Z

QA tests have finished for PR 1958 at commit e3110b9.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2014-08-20T11:08:28Z

Alright, I am merging this.

This patch adds unit tests for Spark Sink. It also removes the private[flume] for Spark Sink, since the sink is instantiated from Flume configuration (looks like this is ignored by reflection which is used by Flume, but we should still remove it anyway). Author: Hari Shreedharan <hshreedharan@apache.org> Author: Hari Shreedharan <hshreedharan@cloudera.com> Closes #1958 from harishreedharan/spark-sink-test and squashes the following commits: e3110b9 [Hari Shreedharan] Add a sleep to allow sink to commit the transactions 120b81e [Hari Shreedharan] Fix complexity in threading model in test 4df5be6 [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into spark-sink-test c9190d1 [Hari Shreedharan] Indentation and spaces changes 7fedc5a [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into spark-sink-test abc20cb [Hari Shreedharan] Minor test changes 7b9b649 [Hari Shreedharan] Merge branch 'master' into spark-sink-test f2c56c9 [Hari Shreedharan] Update SparkSinkSuite.scala a24aac8 [Hari Shreedharan] Remove unused var c86d615 [Hari Shreedharan] [SPARK-3054][STREAMING] Add unit tests for Spark Sink. (cherry picked from commit 8c5a222) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>

tgravescs · 2014-08-20T15:06:17Z

@tdas @harishreedharan This is causing yarn builds to fail on branch-1.1.

I think you are missing commit: 95470a0#diff-74323d4d6986ea21770bf3f49c091e5b

/external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala:66: value getPort is not a member of org.apache.spark.streaming.flume.sink.SparkSink
[ERROR] val port = sink.getPort

harishreedharan · 2014-08-20T17:15:56Z

Correct. This patch uses a method which was introduced in that one.

tgravescs · 2014-08-21T13:15:15Z

I see @pwendell temporarily removed the test: 1d5e84a#diff-d41d8cd98f00b204e9800998ecf8427e

@tdas @harishreedharan can the other commit go in or can you please file jira to followup.

harishreedharan · 2014-08-21T17:37:47Z

I believe the commit which added the getPort() method had merge conflicts in the 1.1 branch, which is why @tdas did not backport it. We can probably backport all commits that went into the flume-sink module to take care of this.

tdas · 2014-08-22T01:34:57Z

I have backported this to branch-1.1. However @pwendell still found some
other mysterious issues with SparkSinkSuite that broke the maven build, so
he deleted sparkSinkSuite temporarily to create another snapshot release of
1.1. I am investigating this right now.

TD

On Thu, Aug 21, 2014 at 10:37 AM, Hari Shreedharan <notifications@github.com

wrote:

I believe the commit which added the getPort() method had merge conflicts
in the 1.1 branch, which is why @tdas https://github.com/tdas did not
backport it. We can probably backport all commits that went into the
flume-sink module to take care of this.

—
Reply to this email directly or view it on GitHub
#1958 (comment).

This patch adds unit tests for Spark Sink. It also removes the private[flume] for Spark Sink, since the sink is instantiated from Flume configuration (looks like this is ignored by reflection which is used by Flume, but we should still remove it anyway). Author: Hari Shreedharan <hshreedharan@apache.org> Author: Hari Shreedharan <hshreedharan@cloudera.com> Closes apache#1958 from harishreedharan/spark-sink-test and squashes the following commits: e3110b9 [Hari Shreedharan] Add a sleep to allow sink to commit the transactions 120b81e [Hari Shreedharan] Fix complexity in threading model in test 4df5be6 [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into spark-sink-test c9190d1 [Hari Shreedharan] Indentation and spaces changes 7fedc5a [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into spark-sink-test abc20cb [Hari Shreedharan] Minor test changes 7b9b649 [Hari Shreedharan] Merge branch 'master' into spark-sink-test f2c56c9 [Hari Shreedharan] Update SparkSinkSuite.scala a24aac8 [Hari Shreedharan] Remove unused var c86d615 [Hari Shreedharan] [SPARK-3054][STREAMING] Add unit tests for Spark Sink.

* rdar://127304026 Update UC-Spark-Authz Plugin to 0.2.1 * rdar://127304026 Update UC-Spark-Authz Plugin to 0.2.2

harishreedharan added 2 commits August 14, 2014 17:19

Remove unused var

a24aac8

jerryshao reviewed Aug 15, 2014
View reviewed changes

Update SparkSinkSuite.scala

f2c56c9

Merge branch 'master' into spark-sink-test

7b9b649

Minor test changes

abc20cb

Merge remote-tracking branch 'asf/master' into spark-sink-test

7fedc5a

tdas reviewed Aug 18, 2014
View reviewed changes

Indentation and spaces changes

c9190d1

tdas reviewed Aug 19, 2014
View reviewed changes

asfgit closed this in 8c5a222 Aug 20, 2014

harishreedharan deleted the spark-sink-test branch September 25, 2014 03:48

szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Aug 7, 2024

rdar://127304026 Update UC-Spark-Authz Plugin to 0.2.2 (apache#1958)

a728c1d

* rdar://127304026 Update UC-Spark-Authz Plugin to 0.2.1 * rdar://127304026 Update UC-Spark-Authz Plugin to 0.2.2



		/*
		* Licensed to the Apache Software Foundation (ASF) under one or more

[SPARK-3054][STREAMING] Add unit tests for Spark Sink. #1958

[SPARK-3054][STREAMING] Add unit tests for Spark Sink. #1958

Uh oh!

Conversation

harishreedharan commented Aug 15, 2014

Uh oh!

SparkQA commented Aug 15, 2014

Uh oh!

SparkQA commented Aug 15, 2014

Uh oh!

jerryshao Aug 15, 2014

Choose a reason for hiding this comment

Uh oh!

harishreedharan Aug 15, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 15, 2014

Uh oh!

harishreedharan commented Aug 15, 2014

Uh oh!

SparkQA commented Aug 15, 2014

Uh oh!

SparkQA commented Aug 15, 2014

Uh oh!

harishreedharan commented Aug 15, 2014

Uh oh!

SparkQA commented Aug 15, 2014

Uh oh!

SparkQA commented Aug 15, 2014

Uh oh!

harishreedharan commented Aug 16, 2014

Uh oh!

harishreedharan commented Aug 16, 2014

Uh oh!

SparkQA commented Aug 18, 2014

Uh oh!

SparkQA commented Aug 18, 2014

Uh oh!

tdas Aug 18, 2014

Choose a reason for hiding this comment

Uh oh!

harishreedharan Aug 18, 2014

Choose a reason for hiding this comment

Uh oh!

tdas Aug 19, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 18, 2014

Uh oh!

SparkQA commented Aug 18, 2014

Uh oh!

tdas commented Aug 19, 2014

Uh oh!

tdas Aug 19, 2014

Choose a reason for hiding this comment

Uh oh!

tdas Aug 19, 2014

Choose a reason for hiding this comment

Uh oh!

harishreedharan commented Aug 19, 2014

Uh oh!

tdas commented Aug 19, 2014

Uh oh!

SparkQA commented Aug 19, 2014

Uh oh!

tdas commented Aug 19, 2014

Uh oh!

harishreedharan commented Aug 19, 2014

Uh oh!

harishreedharan commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014

Uh oh!

harishreedharan commented Aug 20, 2014

Uh oh!

tdas commented Aug 20, 2014

Uh oh!

SparkQA commented Aug 20, 2014