[SPARK-3430] [PySpark] [Doc] generate PySpark API docs using Sphinx #2292

davies · 2014-09-05T19:09:33Z

Using Sphinx to generate API docs for PySpark.

requirement: Sphinx

$ cd python/docs/
$ make html

The generated API docs will be located at python/docs/_build/html/index.html

It can co-exists with those generated by Epydoc.

This is the first working version, after merging in, then we can continue to improve it and replace the epydoc finally.

JoshRosen · 2014-09-05T20:13:27Z

When I tried this, I got a lot of warnings saying that Sphinx couldn't import the PySpark modules:

[joshrosen python (9081ead...)]$ make html
sphinx-build -b html -d _build/doctrees   . _build/html
Running Sphinx v1.2.2
loading pickled environment... done
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 3 changed, 0 removed
reading sources... [100%] pyspark.sql
/Users/joshrosen/Documents/spark/docs/api/python/pyspark.rst:16: WARNING: autodoc: failed to import module u'pyspark'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/joshrosen/anaconda/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
ImportError: No module named pyspark
/Users/joshrosen/Documents/spark/docs/api/python/pyspark.mllib.rst:10: WARNING: autodoc: failed to import module u'pyspark.mllib.classification'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/joshrosen/anaconda/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
ImportError: No module named pyspark.mllib.classification

I guess this is because $SPARK_HOME/python/ isn't on the default PYTHONPATH.

When I do

export PYTHONPATH=../../../python/

it picks up the imports and runs.

davies · 2014-09-05T20:16:52Z

thanks, I had update the PR description.

JoshRosen · 2014-09-05T20:18:30Z

Maybe we can just stick that command inside the Makefile...

JoshRosen · 2014-09-05T20:19:51Z

For those unfamiliar with Sphinx, here's a screenshot of the new docs (which look great!):

JoshRosen · 2014-09-05T20:20:50Z

It looks like there's some markup, like C{(a, b)}, that doesn't render properly since it's a holdover from the epydocs:

JoshRosen · 2014-09-05T20:24:17Z

There are also a few cases of docstrings that didn't get rendered properly:

This looks really good overall. One wishlist item: can we add links from the homepage / index.html to the core classes (RDD, SparkContext, etc.)?

davies · 2014-09-05T20:24:27Z

Yes, I tried to convert most of the markup (but not all of them), what does this mean in epydocs?

JoshRosen · 2014-09-05T20:26:25Z

The markup like C{}, I{}, etc. is epydoc's inline markup. In Sphinx's inline markup, I think you can just use backticks for denoting variables, etc.

nchammas · 2014-09-05T20:29:35Z

There are also a few cases of docstrings that didn't get rendered properly:

FYI: Those ones are broken in epydoc, too, so in the case of top() and others broken in that fashion I doubt it's a Sphinx issue.

JoshRosen · 2014-09-05T20:33:16Z

I guess we have some choices for the markup language dialect that we use for our docstrings. I tend to prefer the Google style to ReStructuredText / Javadoc, since it's a little less cluttered to edit / read. We can revisit this later, though; I don't think it's important to rewrite all of the docstrings as part of this patch.

davies · 2014-09-05T21:20:17Z

@JoshRosen @nchammas I had addressed all the comments, please take a look again.

nchammas · 2014-09-05T21:33:09Z

Oh, sorry I didn't mean the badly formatted doc strings should be fixed in this PR. At least not the ones that are also bad in epydoc. That probably should be left for another PR.

There are many strings that need fixing:

Etc...

JoshRosen · 2014-09-05T21:49:18Z

Hmm, the Makefile PYTHONPATH didn't seem to work. Maybe we should add those directories to sys.path in conf.py, which has a section for this:

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('.'))

davies · 2014-09-05T22:37:30Z

@JoshRosen The Makefile has been fixed.

davies · 2014-09-06T06:23:49Z

Jenkins, test this please.

JoshRosen · 2014-09-06T21:40:35Z

Jenkins, retest this please.

marmbrus · 2014-09-10T02:40:01Z

Jenkins, test this please.

SparkQA · 2014-09-10T02:51:43Z

QA tests have started for PR 2292 at commit 9468ab0.

This patch merges cleanly.

SparkQA · 2014-09-10T04:03:38Z

QA tests have finished for PR 2292 at commit 9468ab0.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2014-09-10T17:36:56Z

Weird, I've never seen Hive fail to initialize like this before.

marmbrus · 2014-09-10T17:37:04Z

Jenkins, test this please.

SparkQA · 2014-09-10T17:52:18Z

QA tests have started for PR 2292 at commit 9468ab0.

This patch merges cleanly.

SparkQA · 2014-09-10T17:53:18Z

QA tests have finished for PR 2292 at commit 9468ab0.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-11T04:33:06Z

QA tests have started for PR 2292 at commit 9468ab0.

This patch merges cleanly.

SparkQA · 2014-09-11T05:33:35Z

QA tests have finished for PR 2292 at commit 9468ab0.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

rnowling · 2014-09-12T13:55:09Z

I actually fixed a number of the Python docstrings previously:
#1808

The changes just didn't show up in the docs until the 1.1.0 release.

I added a new PR to fix .top() :
#2370

Since this PR includes the functionality of my new PR, should I close the new PR?

Conflicts: python/pyspark/serializers.py

SparkQA · 2014-09-13T05:24:22Z

QA tests have started for PR 2292 at commit 1573298.

This patch merges cleanly.

SparkQA · 2014-09-13T05:25:24Z

QA tests have finished for PR 2292 at commit 1573298.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception
- class Dummy(object):

davies · 2014-09-13T05:25:27Z

@JoshRosen I had moved the docs to python/docs/, rebased with master. I think it's ready to merge, please take another look, thanks.

SparkQA · 2014-09-13T05:29:17Z

QA tests have started for PR 2292 at commit 425a3b1.

This patch merges cleanly.

SparkQA · 2014-09-13T07:29:17Z

Tests timed out after a configured wait of 120m.

SparkQA · 2014-09-13T07:31:57Z

QA tests have started for PR 2292 at commit 425a3b1.

This patch merges cleanly.

SparkQA · 2014-09-13T09:31:58Z

Tests timed out after a configured wait of 120m.

SparkQA · 2014-09-13T22:00:54Z

QA tests have started for PR 2292 at commit 425a3b1.

This patch merges cleanly.

SparkQA · 2014-09-13T22:50:59Z

QA tests have finished for PR 2292 at commit 425a3b1.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class JavaSparkContext(val sc: SparkContext)
- class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception
- class Dummy(object):
- class JavaStreamingContext(val ssc: StreamingContext) extends Closeable

SparkQA · 2014-09-14T03:56:47Z

QA tests have started for PR 2292 at commit 425a3b1.

This patch merges cleanly.

SparkQA · 2014-09-14T05:03:31Z

QA tests have finished for PR 2292 at commit 425a3b1.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class JavaSparkContext(val sc: SparkContext)
- throw new IllegalStateException("The main method in the given main class must be static")
- class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception
- class Dummy(object):
- class RatingDeserializer(FramedSerializer):
- case class CreateTableAsSelect(
- class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Encoder extends compression.Encoder[IntegerType.type]
- class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[IntegerType.type])
- class Encoder extends compression.Encoder[LongType.type]
- class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[LongType.type])
- case class CreateTableAsSelect(
- class JavaStreamingContext(val ssc: StreamingContext) extends Closeable

JoshRosen · 2014-09-16T19:51:33Z

This looks good to me; I'm going to merge this into master but leave the JIRA open so that we remember to eventually remove the epydocs / etc.

davies added 2 commits September 5, 2014 11:37

generate PySpark API docs using Sphinx

9081ead

update name and version

e2ccb1b

address all comments

b408f38

fix makefile

9468ab0

davies added 3 commits September 12, 2014 22:16

Merge branch 'master' into sphinx

5fe3903

Conflicts: python/pyspark/serializers.py

move docs to python/docs/

1573298

cleanup

425a3b1

asfgit closed this in ec1adec Sep 16, 2014

[SPARK-3430] [PySpark] [Doc] generate PySpark API docs using Sphinx #2292

[SPARK-3430] [PySpark] [Doc] generate PySpark API docs using Sphinx #2292

Uh oh!

Conversation

davies commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

davies commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

davies commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

nchammas commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

davies commented Sep 5, 2014

Uh oh!

nchammas commented Sep 5, 2014

Uh oh!

JoshRosen commented Sep 5, 2014

Uh oh!

davies commented Sep 5, 2014

Uh oh!

davies commented Sep 6, 2014

Uh oh!

JoshRosen commented Sep 6, 2014

Uh oh!

marmbrus commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 10, 2014

Uh oh!

marmbrus commented Sep 10, 2014

Uh oh!

marmbrus commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 10, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

rnowling commented Sep 12, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

davies commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 13, 2014

Uh oh!

SparkQA commented Sep 14, 2014

Uh oh!

SparkQA commented Sep 14, 2014

Uh oh!