[SPARK-4806] Streaming doc update for 1.2 #3653

tdas · 2014-12-10T00:42:56Z

Important updates to the streaming programming guide

Make the fault-tolerance properties easier to understand, with information about write ahead logs
Update the information about deploying the spark streaming app with information about Driver HA
Update Receiver guide to discuss reliable vs unreliable receivers.

SparkQA · 2014-12-10T00:48:09Z

Test build #24276 has started for PR 3653 at commit a0217c0.

This patch does not merge cleanly.

SparkQA · 2014-12-10T02:26:25Z

Test build #24276 has finished for PR 3653 at commit a0217c0.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-10T02:26:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24276/
Test PASSed.

…c-update-1.2 Conflicts: docs/streaming-programming-guide.md

SparkQA · 2014-12-10T03:15:33Z

Test build #24294 has started for PR 3653 at commit 17b99fb.

This patch merges cleanly.

SparkQA · 2014-12-10T04:34:00Z

Test build #24294 has finished for PR 3653 at commit 17b99fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-10T04:34:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24294/
Test PASSed.

…ploying section, and also updated configuration.

tdas · 2014-12-10T14:27:13Z

@JoshRosen @pwendell @andrewor14

SparkQA · 2014-12-10T14:30:45Z

Test build #24312 has started for PR 3653 at commit 195852c.

This patch merges cleanly.

SparkQA · 2014-12-10T14:37:55Z

Test build #24313 has started for PR 3653 at commit aa8bb87.

This patch merges cleanly.

SparkQA · 2014-12-10T15:55:16Z

Test build #24312 has finished for PR 3653 at commit 195852c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-10T15:55:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24312/
Test PASSed.

SparkQA · 2014-12-10T15:58:50Z

Test build #24313 has finished for PR 3653 at commit aa8bb87.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-10T15:58:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24313/
Test PASSed.

andrewor14 · 2014-12-10T20:46:14Z

Is this before or after @JoshRosen's extensive comments offline?

…when skipping over the headings

Minor edits in Streaming Programming Guide

SparkQA · 2014-12-10T23:57:33Z

Test build #24332 has started for PR 3653 at commit b9c8c24.

This patch merges cleanly.

… streaming-doc-update-1.2 Conflicts: docs/streaming-programming-guide.md

SparkQA · 2014-12-11T00:27:36Z

Test build #24333 has started for PR 3653 at commit 7787209.

This patch merges cleanly.

SparkQA · 2014-12-11T00:49:55Z

Test build #24338 has started for PR 3653 at commit ca19078.

This patch merges cleanly.

SparkQA · 2014-12-11T00:57:36Z

Test build #24340 has started for PR 3653 at commit ce299e4.

This patch merges cleanly.

SparkQA · 2014-12-11T01:04:25Z

Test build #24332 has finished for PR 3653 at commit b9c8c24.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-11T01:04:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24332/
Test FAILed.

JoshRosen · 2014-12-11T01:10:02Z

docs/streaming-programming-guide.md

+
+
+- *[Experimental in Spark 1.2] Configuring write ahead logs* - In Spark 1.2,
+  we have introduced a new experimental feature of write ahead logs for achieved strong


"for achieved" -> "for achieving" or "to achieve"

JoshRosen · 2014-12-11T01:37:31Z

Left a couple of minor comments, but overall this looks good to me.

SparkQA · 2014-12-11T01:53:33Z

Test build #24333 has finished for PR 3653 at commit 7787209.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-11T01:53:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24333/
Test PASSed.

SparkQA · 2014-12-11T02:16:59Z

Test build #24338 has finished for PR 3653 at commit ca19078.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-11T02:17:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24338/
Test PASSed.

SparkQA · 2014-12-11T02:24:39Z

Test build #24340 has finished for PR 3653 at commit ce299e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-11T02:24:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24340/
Test PASSed.

Important updates to the streaming programming guide - Make the fault-tolerance properties easier to understand, with information about write ahead logs - Update the information about deploying the spark streaming app with information about Driver HA - Update Receiver guide to discuss reliable vs unreliable receivers. Author: Tathagata Das <tathagata.das1565@gmail.com> Author: Josh Rosen <joshrosen@databricks.com> Author: Josh Rosen <rosenville@gmail.com> Closes #3653 from tdas/streaming-doc-update-1.2 and squashes the following commits: f53154a [Tathagata Das] Addressed Josh's comments. ce299e4 [Tathagata Das] Minor update. ca19078 [Tathagata Das] Minor change f746951 [Tathagata Das] Mentioned performance problem with WAL 7787209 [Tathagata Das] Merge branch 'streaming-doc-update-1.2' of github.com:tdas/spark into streaming-doc-update-1.2 2184729 [Tathagata Das] Updated Kafka and Flume guides with reliability information. 2f3178c [Tathagata Das] Added more information about writing reliable receivers in the custom receiver guide. 91aa5aa [Tathagata Das] Improved API Docs menu 5707581 [Tathagata Das] Added Pythn API badge b9c8c24 [Tathagata Das] Merge pull request #26 from JoshRosen/streaming-programming-guide b8c8382 [Josh Rosen] minor fixes a4ef126 [Josh Rosen] Restructure parts of the fault-tolerance section to read a bit nicer when skipping over the headings 65f66cd [Josh Rosen] Fix broken link to fault-tolerance semantics section. f015397 [Josh Rosen] Minor grammar / pluralization fixes. 3019f3a [Josh Rosen] Fix minor Markdown formatting issues aa8bb87 [Tathagata Das] Small update. 195852c [Tathagata Das] Updated based on Josh's comments, updated receiver reliability and deploying section, and also updated configuration. 17b99fb [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into streaming-doc-update-1.2 a0217c0 [Tathagata Das] Changed Deploying menu layout 67fcffc [Tathagata Das] Added cluster mode + supervise example to submitting application guide. e45453b [Tathagata Das] Update streaming guide, added deploying section. 192c7a7 [Tathagata Das] Added more info about Python API, and rewrote the checkpointing section. (cherry picked from commit b004150) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>

tdas · 2014-12-11T14:32:09Z

@JoshRosen I have addressed your final comments and merged it. Thank you very much.

AmplabJenkins · 2014-12-11T14:37:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24370/
Test FAILed.

tdas added 4 commits December 7, 2014 22:08

Added more info about Python API, and rewrote the checkpointing section.

192c7a7

Update streaming guide, added deploying section.

e45453b

Added cluster mode + supervise example to submitting application guide.

67fcffc

Changed Deploying menu layout

a0217c0

Merge remote-tracking branch 'apache-github/master' into streaming-do…

17b99fb

…c-update-1.2 Conflicts: docs/streaming-programming-guide.md

Updated based on Josh's comments, updated receiver reliability and de…

195852c

…ploying section, and also updated configuration.

Small update.

aa8bb87

JoshRosen and others added 6 commits December 10, 2014 14:27

Fix minor Markdown formatting issues

3019f3a

Minor grammar / pluralization fixes.

f015397

Fix broken link to fault-tolerance semantics section.

65f66cd

Restructure parts of the fault-tolerance section to read a bit nicer …

a4ef126

…when skipping over the headings

minor fixes

b8c8382

Merge pull request #26 from JoshRosen/streaming-programming-guide

b9c8c24

Minor edits in Streaming Programming Guide

tdas added 2 commits December 10, 2014 16:12

Added Pythn API badge

5707581

Improved API Docs menu

91aa5aa

Merge branch 'streaming-doc-update-1.2' of github.com:tdas/spark into…

7787209

… streaming-doc-update-1.2 Conflicts: docs/streaming-programming-guide.md

tdas added 2 commits December 10, 2014 16:41

Mentioned performance problem with WAL

f746951

Minor change

ca19078

Minor update.

ce299e4

JoshRosen reviewed Dec 11, 2014
View reviewed changes

Addressed Josh's comments.

f53154a

asfgit closed this in b004150 Dec 11, 2014



		- [Experimental in Spark 1.2] Configuring write ahead logs - In Spark 1.2,
		we have introduced a new experimental feature of write ahead logs for achieved strong

[SPARK-4806] Streaming doc update for 1.2 #3653

[SPARK-4806] Streaming doc update for 1.2 #3653

Uh oh!

Conversation

tdas commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

AmplabJenkins commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

AmplabJenkins commented Dec 10, 2014

Uh oh!

tdas commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

AmplabJenkins commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

AmplabJenkins commented Dec 10, 2014

Uh oh!

andrewor14 commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 11, 2014

Uh oh!

SparkQA commented Dec 11, 2014

Uh oh!

SparkQA commented Dec 11, 2014

Uh oh!

SparkQA commented Dec 11, 2014

Uh oh!

AmplabJenkins commented Dec 11, 2014

Uh oh!

JoshRosen Dec 11, 2014

Choose a reason for hiding this comment

Uh oh!

JoshRosen commented Dec 11, 2014

Uh oh!

SparkQA commented Dec 11, 2014

Uh oh!

AmplabJenkins commented Dec 11, 2014

Uh oh!

SparkQA commented Dec 11, 2014

Uh oh!

AmplabJenkins commented Dec 11, 2014

Uh oh!

SparkQA commented Dec 11, 2014

Uh oh!

AmplabJenkins commented Dec 11, 2014

Uh oh!

tdas commented Dec 11, 2014

Uh oh!

AmplabJenkins commented Dec 11, 2014

Uh oh!

Uh oh!