[SPARK-37474][R][DOCS] Migrate SparkR docs to pkgdown #34728

+# Determine Spark(R) version
+SPARK_VERSION=$(grep -oP "(?<=Version:\ ).*" ../DESCRIPTION)
+
+# Update url
+sed "s/{SPARK_VERSION}/$SPARK_VERSION/" ../pkgdown/_pkgdown_template.yml > ../_pkgdown.yml
+
+"$R_SCRIPT_PATH/Rscript" -e 'libDir <- "../../lib"; library(SparkR, lib.loc=libDir); pkgdown::build_site("..")'
+
+# Clean temporary config
+rm ../_pkgdown.yml
+


I am not sure if that's the best way to handle different versions, so any suggestions are welcome.

SparkQA · 2021-12-06T14:37:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50429/

SparkQA · 2021-12-06T15:19:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50429/

HyukjinKwon · 2021-12-07T00:20:08Z

.github/workflows/build_and_test.yml

@@ -489,6 +490,8 @@ jobs:
        apt-get update -y
        apt-get install -y ruby ruby-dev
        Rscript -e "install.packages(c('devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2'), repos='https://cloud.r-project.org/')"
+        Rscript -e "devtools::install_git('https://github.com/r-lib/pkgdown.git')"
+        Rscript -e "devtools::install_git('https://github.com/amirmasoudabdol/preferably.git')"


quick question, do we need this for documentation build? we would have to also update https://github.com/apache/spark/blob/master/docs/README.md and https://github.com/apache/spark/blob/master/dev/create-release/spark-rm/Dockerfile#L83-L84 . Can be done separately though.

And can we pin the versions of both libraries instead of using the master branch?

quick question, do we need this for documentation build?

Yes, documentation build invokes pkgdown.

And can we pin the versions of both libraries instead of using the master branch?

Of course. We should be even able to pin CRAN version of pkgdown (seems like 2.0.x and compatible preferably)) are already published and visible).

R/pkg/DESCRIPTION

R/pkg/vignettes/sparkr-vignettes.Rmd

HyukjinKwon · 2021-12-07T00:22:41Z

Looks pretty good .. but the packaging stuff it would be great to have sign-offs from @felixcheung @shivaram @mengxr or @falaki.

SparkQA · 2021-12-07T15:28:25Z

Test build #145990 has finished for PR 34728 at commit 358cc5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-07T15:33:55Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50466/

SparkQA · 2021-12-07T16:31:53Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50466/

SparkQA · 2021-12-07T18:42:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50467/

SparkQA · 2021-12-07T19:41:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50467/

SparkQA · 2021-12-07T20:34:34Z

Test build #145991 has finished for PR 34728 at commit 6e01b24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

LGTM

srowen · 2021-12-10T14:37:01Z

Merged to master

### What changes were proposed in this pull request? This tries to fix `do-release-docker.sh` for branch-3.2. ### Why are the changes needed? Currently the following error will occur if running the script in `branch-3.2`: ``` #5 917.4 g++ -std=gnu++14 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o testthat.so init.o reassign.o test-catch.o test-example.o test-runner.o -L/usr/lib/R/lib -lR #5 917.5 installing to /usr/local/lib/R/site-library/00LOCK-testthat/00new/testthat/libs #5 917.5 ** R #5 917.5 ** inst #5 917.5 ** byte-compile and prepare package for lazy loading #5 924.4 ** help #5 924.6 *** installing help indices #5 924.7 *** copying figures #5 924.7 ** building package indices #5 924.9 ** installing vignettes #5 924.9 ** testing if installed package can be loaded from temporary location #5 925.1 ** checking absolute paths in shared objects and dynamic libraries #5 925.1 ** testing if installed package can be loaded from final location #5 925.5 ** testing if installed package keeps a record of temporary installation path #5 925.5 * DONE (testthat) #5 925.8 ERROR: dependency 'pkgdown' is not available for package 'devtools' #5 925.8 * removing '/usr/local/lib/R/site-library/devtools' #5 925.8 #5 925.8 The downloaded source packages are in #5 925.8 '/tmp/Rtmp3nJI60/downloaded_packages' #5 925.8 Warning messages: #5 925.8 1: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : #5 925.8 installation of package 'textshaping' had non-zero exit status #5 925.8 2: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : #5 925.8 installation of package 'ragg' had non-zero exit status #5 925.8 3: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : #5 925.8 installation of package 'pkgdown' had non-zero exit status #5 925.8 4: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : #5 925.8 installation of package 'devtools' had non-zero exit status #5 926.0 Error in loadNamespace(x) : there is no package called 'devtools' #5 926.0 Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart #5 926.0 Execution halted ``` The same error doesn't happen on master. I checked the diff between the two and it seems the following line: ``` $APT_INSTALL libfontconfig1-dev libharfbuzz-dev libfribidi-dev libfreetype6-dev libpng-dev libtiff5-dev libjpeg-dev && \ ``` introduced in #34728 made the difference. I verified that after adding the line, `do-release-docker.sh` (dry run mode) was able to finish successfully. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually Closes #38643 from sunchao/fix-docker-release. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This tries to fix `do-release-docker.sh` for branch-3.2. ### Why are the changes needed? Currently the following error will occur if running the script in `branch-3.2`: ``` apache#5 917.4 g++ -std=gnu++14 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o testthat.so init.o reassign.o test-catch.o test-example.o test-runner.o -L/usr/lib/R/lib -lR apache#5 917.5 installing to /usr/local/lib/R/site-library/00LOCK-testthat/00new/testthat/libs apache#5 917.5 ** R apache#5 917.5 ** inst apache#5 917.5 ** byte-compile and prepare package for lazy loading apache#5 924.4 ** help apache#5 924.6 *** installing help indices apache#5 924.7 *** copying figures apache#5 924.7 ** building package indices apache#5 924.9 ** installing vignettes apache#5 924.9 ** testing if installed package can be loaded from temporary location apache#5 925.1 ** checking absolute paths in shared objects and dynamic libraries apache#5 925.1 ** testing if installed package can be loaded from final location apache#5 925.5 ** testing if installed package keeps a record of temporary installation path apache#5 925.5 * DONE (testthat) apache#5 925.8 ERROR: dependency 'pkgdown' is not available for package 'devtools' apache#5 925.8 * removing '/usr/local/lib/R/site-library/devtools' apache#5 925.8 apache#5 925.8 The downloaded source packages are in apache#5 925.8 '/tmp/Rtmp3nJI60/downloaded_packages' apache#5 925.8 Warning messages: apache#5 925.8 1: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : apache#5 925.8 installation of package 'textshaping' had non-zero exit status apache#5 925.8 2: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : apache#5 925.8 installation of package 'ragg' had non-zero exit status apache#5 925.8 3: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : apache#5 925.8 installation of package 'pkgdown' had non-zero exit status apache#5 925.8 4: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat", : apache#5 925.8 installation of package 'devtools' had non-zero exit status apache#5 926.0 Error in loadNamespace(x) : there is no package called 'devtools' apache#5 926.0 Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart apache#5 926.0 Execution halted ``` The same error doesn't happen on master. I checked the diff between the two and it seems the following line: ``` $APT_INSTALL libfontconfig1-dev libharfbuzz-dev libfribidi-dev libfreetype6-dev libpng-dev libtiff5-dev libjpeg-dev && \ ``` introduced in apache#34728 made the difference. I verified that after adding the line, `do-release-docker.sh` (dry run mode) was able to finish successfully. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually Closes apache#38643 from sunchao/fix-docker-release. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added DOCS R SQL labels Nov 27, 2021

github-actions bot added the INFRA label Nov 28, 2021

srowen reviewed Nov 28, 2021

View reviewed changes

R/pkg/vignettes/sparkr-vignettes.Rmd Outdated Show resolved Hide resolved

zero323 commented Nov 28, 2021

View reviewed changes

R/pkg/_pkgdown.yml Outdated Show resolved Hide resolved

Generate URL based on the package version

a18cc0c

zero323 commented Dec 6, 2021

View reviewed changes

zero323 changed the title ~~[WIP][SPARK-37474][R][DOCS] Migrate SparkR docs to pkgdown~~ [SPARK-37474][R][DOCS] Migrate SparkR docs to pkgdown Dec 6, 2021

zero323 marked this pull request as ready for review December 6, 2021 14:46

HyukjinKwon reviewed Dec 7, 2021

View reviewed changes

R/pkg/DESCRIPTION Outdated Show resolved Hide resolved

HyukjinKwon reviewed Dec 7, 2021

View reviewed changes

R/pkg/vignettes/sparkr-vignettes.Rmd Outdated Show resolved Hide resolved

zero323 added 3 commits December 7, 2021 15:14

JVm -> JVM

b353bee

Revert RoxygenNote change

156ed2c

Pin pkgdown and preferably versions

358cc5c

Update release image and docs README.md

6e01b24

github-actions bot added the BUILD label Dec 7, 2021

HyukjinKwon approved these changes Dec 8, 2021

View reviewed changes

srowen closed this in 16d1c68 Dec 10, 2021

sunchao mentioned this pull request Nov 14, 2022

[SPARK-41091][BUILD][3.2] Fix Docker release tool for branch-3.2 #38643

Closed

		reference:

[SPARK-37474][R][DOCS] Migrate SparkR docs to pkgdown #34728

[SPARK-37474][R][DOCS] Migrate SparkR docs to pkgdown #34728

Conversation

zero323 commented Nov 27, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 27, 2021

SparkQA commented Nov 28, 2021

HyukjinKwon commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

srowen left a comment

Choose a reason for hiding this comment

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

zero323 Nov 28, 2021

Choose a reason for hiding this comment

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Nov 28, 2021

SparkQA commented Dec 6, 2021

SparkQA commented Dec 6, 2021

zero323 Dec 6, 2021

Choose a reason for hiding this comment

SparkQA commented Dec 6, 2021

SparkQA commented Dec 6, 2021

HyukjinKwon Dec 7, 2021

Choose a reason for hiding this comment

zero323 Dec 7, 2021

Choose a reason for hiding this comment

HyukjinKwon commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

HyukjinKwon left a comment

Choose a reason for hiding this comment

srowen commented Dec 10, 2021

zero323 commented Nov 27, 2021 •

edited

Loading