You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/README.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
Welcome to the Spark documentation!
2
2
3
-
This readme will walk you through navigating and building the Spark documentation, which is included here with the Spark source code. You can also find documentation specific to release versions of Spark at http://spark.incubator.apache.org/documentation.html.
3
+
This readme will walk you through navigating and building the Spark documentation, which is included here with the Spark source code. You can also find documentation specific to release versions of Spark at http://spark.apache.org/documentation.html.
4
4
5
5
Read on to learn more about viewing documentation in plain text (i.e., markdown) or building the documentation yourself. Why build it yourself? So that you have the docs that corresponds to whichever version of Spark you currently have checked out of revision control.
Copy file name to clipboardExpand all lines: docs/bagel-programming-guide.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -108,7 +108,7 @@ _Example_
108
108
109
109
## Operations
110
110
111
-
Here are the actions and types in the Bagel API. See [Bagel.scala](https://github.com/apache/incubator-spark/blob/master/bagel/src/main/scala/org/apache/spark/bagel/Bagel.scala) for details.
111
+
Here are the actions and types in the Bagel API. See [Bagel.scala](https://github.com/apache/spark/blob/master/bagel/src/main/scala/org/apache/spark/bagel/Bagel.scala) for details.
Copy file name to clipboardExpand all lines: docs/index.md
+6-6
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ It also supports a rich set of higher-level tools including [Shark](http://shark
9
9
10
10
# Downloading
11
11
12
-
Get Spark by visiting the [downloads page](http://spark.incubator.apache.org/downloads.html) of the Apache Spark site. This documentation is for Spark version {{site.SPARK_VERSION}}.
12
+
Get Spark by visiting the [downloads page](http://spark.apache.org/downloads.html) of the Apache Spark site. This documentation is for Spark version {{site.SPARK_VERSION}}.
13
13
14
14
Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). All you need to run it is to have `java` to installed on your system `PATH`, or the `JAVA_HOME` environment variable pointing to a Java installation.
15
15
@@ -96,7 +96,7 @@ For this version of Spark (0.8.1) Hadoop 2.2.x (or newer) users will have to bui
96
96
*[Amazon EC2](ec2-scripts.html): scripts that let you launch a cluster on EC2 in about 5 minutes
97
97
*[Standalone Deploy Mode](spark-standalone.html): launch a standalone cluster quickly without a third-party cluster manager
98
98
*[Mesos](running-on-mesos.html): deploy a private cluster using
99
-
[Apache Mesos](http://incubator.apache.org/mesos)
99
+
[Apache Mesos](http://mesos.apache.org)
100
100
*[YARN](running-on-yarn.html): deploy Spark on top of Hadoop NextGen (YARN)
101
101
102
102
**Other documents:**
@@ -110,20 +110,20 @@ For this version of Spark (0.8.1) Hadoop 2.2.x (or newer) users will have to bui
*[Shark](http://shark.cs.berkeley.edu): Apache Hive over Spark
115
-
*[Mailing Lists](http://spark.incubator.apache.org/mailing-lists.html): ask questions about Spark here
115
+
*[Mailing Lists](http://spark.apache.org/mailing-lists.html): ask questions about Spark here
116
116
*[AMP Camps](http://ampcamp.berkeley.edu/): a series of training camps at UC Berkeley that featured talks and
117
117
exercises about Spark, Shark, Mesos, and more. [Videos](http://ampcamp.berkeley.edu/agenda-2012),
118
118
[slides](http://ampcamp.berkeley.edu/agenda-2012) and [exercises](http://ampcamp.berkeley.edu/exercises-2012) are
119
119
available online for free.
120
-
*[Code Examples](http://spark.incubator.apache.org/examples.html): more are also available in the [examples subfolder](https://github.com/apache/incubator-spark/tree/master/examples/src/main/scala/) of Spark
120
+
*[Code Examples](http://spark.apache.org/examples.html): more are also available in the [examples subfolder](https://github.com/apache/spark/tree/master/examples/src/main/scala/) of Spark
To get help using Spark or keep up with Spark development, sign up for the [user mailing list](http://spark.incubator.apache.org/mailing-lists.html).
126
+
To get help using Spark or keep up with Spark development, sign up for the [user mailing list](http://spark.apache.org/mailing-lists.html).
127
127
128
128
If you're in the San Francisco Bay Area, there's a regular [Spark meetup](http://www.meetup.com/spark-users/) every few weeks. Come by to meet the developers and other users.
Copy file name to clipboardExpand all lines: docs/java-programming-guide.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -189,7 +189,7 @@ We hope to generate documentation with Java-style syntax in the future.
189
189
# Where to Go from Here
190
190
191
191
Spark includes several sample programs using the Java API in
192
-
[`examples/src/main/java`](https://github.com/apache/incubator-spark/tree/master/examples/src/main/java/org/apache/spark/examples). You can run them by passing the class name to the
192
+
[`examples/src/main/java`](https://github.com/apache/spark/tree/master/examples/src/main/java/org/apache/spark/examples). You can run them by passing the class name to the
193
193
`bin/run-example` script included in Spark; for example:
Copy file name to clipboardExpand all lines: docs/python-programming-guide.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -157,7 +157,7 @@ some example applications.
157
157
158
158
# Where to Go from Here
159
159
160
-
PySpark also includes several sample programs in the [`python/examples` folder](https://github.com/apache/incubator-spark/tree/master/python/examples).
160
+
PySpark also includes several sample programs in the [`python/examples` folder](https://github.com/apache/spark/tree/master/python/examples).
161
161
You can run them by passing the files to `pyspark`; e.g.:
Copy file name to clipboardExpand all lines: docs/scala-programming-guide.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -365,7 +365,7 @@ res2: Int = 10
365
365
366
366
# Where to Go from Here
367
367
368
-
You can see some [example Spark programs](http://spark.incubator.apache.org/examples.html) on the Spark website.
368
+
You can see some [example Spark programs](http://spark.apache.org/examples.html) on the Spark website.
369
369
In addition, Spark includes several samples in `examples/src/main/scala`. Some of them have both Spark versions and local (non-parallel) versions, allowing you to see what had to be changed to make the program run on a cluster. You can run them using by passing the class name to the `bin/run-example` script included in Spark; for example:
Copy file name to clipboardExpand all lines: docs/spark-debugger.md
+2-2
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
layout: global
3
3
title: The Spark Debugger
4
4
---
5
-
**Summary:** The Spark debugger provides replay debugging for deterministic (logic) errors in Spark programs. It's currently in development, but you can try it out in the [arthur branch](https://github.com/apache/incubator-spark/tree/arthur).
5
+
**Summary:** The Spark debugger provides replay debugging for deterministic (logic) errors in Spark programs. It's currently in development, but you can try it out in the [arthur branch](https://github.com/apache/spark/tree/arthur).
6
6
7
7
## Introduction
8
8
@@ -19,7 +19,7 @@ For deterministic errors, debugging a Spark program is now as easy as debugging
19
19
20
20
## Approach
21
21
22
-
As your Spark program runs, the slaves report key events back to the master -- for example, RDD creations, RDD contents, and uncaught exceptions. (A full list of event types is in [EventLogging.scala](https://github.com/apache/incubator-spark/blob/arthur/core/src/main/scala/spark/EventLogging.scala).) The master logs those events, and you can load the event log into the debugger after your program is done running.
22
+
As your Spark program runs, the slaves report key events back to the master -- for example, RDD creations, RDD contents, and uncaught exceptions. (A full list of event types is in [EventLogging.scala](https://github.com/apache/spark/blob/arthur/core/src/main/scala/spark/EventLogging.scala).) The master logs those events, and you can load the event log into the debugger after your program is done running.
23
23
24
24
_A note on nondeterminism:_ For fault recovery, Spark requires RDD transformations (for example, the function passed to `RDD.map`) to be deterministic. The Spark debugger also relies on this property, and it can also warn you if your transformation is nondeterministic. This works by checksumming the contents of each RDD and comparing the checksums from the original execution to the checksums after recomputing the RDD in the debugger.
0 commit comments