[SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page #32546

itholic · 2021-05-14T04:58:58Z

What changes were proposed in this pull request?

This PR proposes move ORC data source options from Python, Scala and Java into a single page.

Why are the changes needed?

So far, the documentation for ORC data source options is separated into different pages for each language API documents. However, this makes managing many options inconvenient, so it is efficient to manage all options in a single page and provide a link to that page in the API of each language.

Does this PR introduce any user-facing change?

Yes, the documents will be shown below after this change:

"ORC Files" page
Python
Scala
Java

How was this patch tested?

Manually build docs and confirm the page.

…a into a single page

SparkQA · 2021-05-14T06:22:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43067/

SparkQA · 2021-05-14T06:22:02Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43067/

sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala

docs/sql-data-sources-orc.md

SparkQA · 2021-05-14T10:09:52Z

Test build #138548 has finished for PR 32546 at commit 0ce01d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-17T12:47:28Z

Test build #138632 has finished for PR 32546 at commit 005d87d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-17T13:27:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43151/

SparkQA · 2021-05-17T13:33:11Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43151/

SparkQA · 2021-05-17T15:25:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43155/

SparkQA · 2021-05-17T15:25:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43155/

SparkQA · 2021-05-17T18:57:08Z

Test build #138635 has finished for PR 32546 at commit aa54b45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…5395

SparkQA · 2021-05-18T01:51:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43175/

SparkQA · 2021-05-18T02:26:24Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43175/

itholic · 2021-05-18T03:22:31Z

Jenkins, retest this please

SparkQA · 2021-05-18T04:26:06Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43182/

SparkQA · 2021-05-18T05:03:08Z

Test build #138654 has finished for PR 32546 at commit 3740c52.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UpdatingSessionsExec(
class UpdatingSessionsIterator(

SparkQA · 2021-05-20T11:29:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43261/

SparkQA · 2021-05-20T12:47:52Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43272/

SparkQA · 2021-05-20T13:06:28Z

Test build #138738 has finished for PR 32546 at commit 0b0e183.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-20T13:18:15Z

Test build #138742 has finished for PR 32546 at commit 6358d59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-20T14:19:18Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43272/

SparkQA · 2021-05-20T16:27:34Z

Test build #138750 has finished for PR 32546 at commit 3971dd8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

python/pyspark/sql/readwriter.py

python/pyspark/sql/streaming.py

HyukjinKwon · 2021-05-21T01:10:12Z

Looks pretty good otherwise. Don't forgot to update Pr description as well. cc @dongjoon-hyun FYI

SparkQA · 2021-05-21T03:02:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43300/

SparkQA · 2021-05-21T03:40:32Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43300/

dongjoon-hyun · 2021-05-21T03:42:14Z

Thank you for pinging me, @HyukjinKwon .

dongjoon-hyun · 2021-05-21T03:45:43Z

docs/sql-data-sources-orc.md

+    <td>write</td>
+  </tr>
+</table>
+Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html"> Generic File Source Options</a>.


Although I know that this is inherited, https://spark.apache.org/docs/latest/ looks fragile to me because it is going to be a broken link when we cut branch-3.2 on July 1st. In branch-3.2, it should point 3.2 document only. Shall we use a relative link instead of /latest/?

Like this PR, we don't know what refactoring happens in the future.

Thanks, @dongjoon-hyun .
I took a look for that but seems tricky to create a link for each release in Scaladoc ..
I created a JIRA to track it separately here: SPARK-35481.
I will take a separate look if that's fine to you too!

dongjoon-hyun · 2021-05-21T03:46:33Z

python/pyspark/sql/readwriter.py

+        ----------------
+        Extra options
+            For the extra options, refer to
+            `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option>`_  # noqa


Ditto. Can we have a more robust link here?

dongjoon-hyun · 2021-05-21T03:48:06Z

sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

-   * </ul>
+   * ORC-specific option(s) for reading ORC files can be found in
+   * <a href=
+   *   "https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option">


dongjoon-hyun · 2021-05-21T03:49:14Z

Thank you, @itholic and @HyukjinKwon . The refactoring idea looks good to me. I commented only a technical issue about the link usage. I'll leave this to @HyukjinKwon .

HyukjinKwon

Okay looks fine with https://github.com/apache/spark/pull/32546/files#r636625094. Please update Pr description.

itholic · 2021-05-21T06:54:45Z

Thanks, @HyukjinKwon .
PR description is updated, and also the PR description of #32204, #32161 are updated as well.

SparkQA · 2021-05-21T07:30:18Z

Test build #138776 has finished for PR 32546 at commit 043d308.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-05-21T09:03:49Z

Merged to master.

HyukjinKwon · 2021-05-25T11:00:18Z

docs/sql-data-sources-orc.md

+  <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+  <tr>
+    <td><code>mergeSchema</code></td>
+    <td>None</td>


@itholic it has the same issue. The default value isn't None but false.

HyukjinKwon · 2021-05-25T11:01:21Z

docs/sql-data-sources-orc.md

+  </tr>
+  <tr>
+    <td><code>compression</code></td>
+    <td>None</td>


In this case when the default value doesn't exist, you can follow https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration (none).

HyukjinKwon · 2021-05-25T11:09:24Z

docs/sql-data-sources-orc.md

+  *  `DataFrameReader`
+  *  `DataFrameWriter`
+  *  `DataStreamReader`
+  *  `DataStreamWriter`


also mention:

* `OPTIONS` clause at [CREATE TABLE USING DATA_SOURCE](sql-ref-syntax-ddl-create-table-datasource.html)

itholic added 2 commits May 14, 2021 10:57

[SPARK-35395][DOCS] Move ORC data source options from Python and Scal…

e9d4605

…a into a single page

Restore mode partitionBy

0ce01d0

github-actions bot added CORE DOCS PYTHON SQL STRUCTURED STREAMING labels May 14, 2021

HyukjinKwon reviewed May 14, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed May 14, 2021

View reviewed changes

docs/sql-data-sources-orc.md Outdated Show resolved Hide resolved

itholic added 4 commits May 17, 2021 21:23

Resolved comments

4b3aa58

Restore unrelated changes

f29ae05

Restore the un related changes

7bba907

Fix comments

005d87d

Resolved Scala linter

aa54b45

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

3740c52

…5395

Retest

2e0787c

itholic added 3 commits May 20, 2021 20:46

itemize the options

69b6d84

Resolved comments

0df8b04

Resolved comments

3971dd8

HyukjinKwon reviewed May 21, 2021

View reviewed changes

python/pyspark/sql/readwriter.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed May 21, 2021

View reviewed changes

python/pyspark/sql/streaming.py Outdated Show resolved Hide resolved

Resolved comments

043d308

dongjoon-hyun reviewed May 21, 2021

View reviewed changes

HyukjinKwon approved these changes May 21, 2021

View reviewed changes

HyukjinKwon closed this in 0fe65b5 May 21, 2021

HyukjinKwon reviewed May 25, 2021

View reviewed changes

HyukjinKwon mentioned this pull request May 26, 2021

[SPARK-35509][DOCS] Move text data source options from Python and Scala into a single page #32660

Closed

[SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page #32546

[SPARK-35395][DOCS] Move ORC data source options from Python and Scala into a single page #32546

Uh oh!

Conversation

itholic commented May 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented May 14, 2021

Uh oh!

SparkQA commented May 14, 2021

Uh oh!

Uh oh!

Uh oh!

SparkQA commented May 14, 2021

Uh oh!

SparkQA commented May 17, 2021

Uh oh!

SparkQA commented May 17, 2021

Uh oh!

SparkQA commented May 17, 2021

Uh oh!

SparkQA commented May 17, 2021

Uh oh!

SparkQA commented May 17, 2021

Uh oh!

SparkQA commented May 17, 2021

Uh oh!

SparkQA commented May 18, 2021

Uh oh!

SparkQA commented May 18, 2021

Uh oh!

itholic commented May 18, 2021

Uh oh!

SparkQA commented May 18, 2021

Uh oh!

SparkQA commented May 18, 2021

Uh oh!

SparkQA commented May 20, 2021

Uh oh!

SparkQA commented May 20, 2021

Uh oh!

SparkQA commented May 20, 2021

Uh oh!

SparkQA commented May 20, 2021

Uh oh!

SparkQA commented May 20, 2021

Uh oh!

SparkQA commented May 20, 2021

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented May 21, 2021

Uh oh!

SparkQA commented May 21, 2021

Uh oh!

SparkQA commented May 21, 2021

Uh oh!

dongjoon-hyun commented May 21, 2021

Uh oh!

dongjoon-hyun May 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itholic May 21, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun May 21, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun May 21, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon left a comment

itholic commented May 14, 2021 •

edited

Loading

dongjoon-hyun May 21, 2021 •

edited

Loading

dongjoon-hyun commented May 21, 2021 •

edited

Loading

itholic commented May 21, 2021 •

edited

Loading

HyukjinKwon May 25, 2021 •

edited

Loading