-
Notifications
You must be signed in to change notification settings - Fork 832
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Wire in the python doc build and related runme work
* Change to making a version string: it used to read an existing
version before re-creating `BuildArtifacts`, so the old version would
still get used. This is now different: the version is properly
re-made when the directory is recreated. This will have a negative
impact on quick builds after a commit, but without it things can be
confusing when checking out a different git branch.
* Pip-install `sphinx`, `sphinx_rtd_theme`, and `recommonmark` in
developer environments for the sphinx doc build.
* Add a build script for the python docs, and wire it into runme.
* Make the generated rst files go in a different BuildArtifacts
directory, so the actual docs that should be copied are all in a
single directory.
* Drop doc references that don't exist.
* Some tweaking of `conf.py`.
* Streamline the build a bit, making the python work more similar to the
Scala work and more:
- always create the pip package instead of when `should publish
pip` (which now controls only the actual publishing);
- do that with the pydocs step right after the sbt step, so there's a
single chunk of python work;
- move the python tests to after the basic publishing, making it
possible to grab the results for debugging in case of failures.
* Add `$MML_LATEST` which is set to `yes` only when we're building a
tagged version *and* that version is the most recent one on `master`.
Use it to determine whether we should label the current build as
latest (for docker hub tag and published docs directory).
* New `azblob` convenience function for blob operations in
`$MAIN_CONTAINER`.
* New `docs` tag for publishing, and new `$DOCS_CONTAINER`, `$DOCS_URL`
environment variables.
* Add a `pydocs/publish` script (not really "py") to copy the generated
documentation to the `docs` container. This is a complete hack
because of several limitations of the CLI, and because it's making
`foo/` URLs work (copied from `foo/index.html`) and also `foo` (a
redirection HTML to `foo/`).
* Add `docs` to what `container-gc` knows about. (But deleting a
version is likely going to be horrifyingly slow.)
* Add `src/project/top-doc.html` with some HTML to show on the toplevel
initial page.- Loading branch information
1 parent
cf8fb19
commit 882f4c2
Showing
16 changed files
with
279 additions
and
76 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| These are the API documentation pages for the Scala side of | ||
| <a href="https://github.com/Azure/mmlspark/">MMLSpark</a>. | ||
|
|
||
| See also the <a href="../pyspark/" target="_parent">pyspark level API documentation</a>. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| #!/usr/bin/env bash | ||
| # Copyright (C) Microsoft Corporation. All rights reserved. | ||
| # Licensed under the MIT License. See LICENSE in project root for information. | ||
|
|
||
| . "$(dirname "${BASH_SOURCE[0]}")/../../runme" "$@" | ||
| main() ( | ||
|
|
||
| show section "Building Python Documentation" | ||
|
|
||
| here="$(cd $(dirname "${BASH_SOURCE[0]}"); pwd)" | ||
| pydocsrc="$BUILD_ARTIFACTS/pydocsrc" | ||
|
|
||
| cd "$BUILD_ARTIFACTS/docs" | ||
| _rm "pyspark"; _md "pyspark" | ||
|
|
||
| show - "Copying source files" | ||
| srcs=( "$here/src"/* ) | ||
| for f in "${srcs[@]}"; do t="$pydocsrc/$(basename "$f")" | ||
| if [[ -e "$t" ]]; then failwith "target file exists: $t"; else cp "$f" "$t"; fi | ||
| done | ||
|
|
||
| PATH+=":$CONDA_HOME/bin" | ||
| PYTHONPATH="$BUILD_ARTIFACTS/sdk/mmlspark.zip:$PYTHONPATH" | ||
| PYTHONPATH="$BASEDIR/src/src/main/resources/mmlspark:$PYTHONPATH" | ||
|
|
||
| _ -a sphinx-build -D version="${MML_VERSION%%+*}" -D release="$MML_VERSION" \ | ||
| "$pydocsrc" "pyspark" | ||
| ret=$? | ||
|
|
||
| for f in "${srcs[@]}"; do rm -f "$pydocsrc/$(basename "$f")"; done | ||
|
|
||
| if (($ret)); then failwith "documentation build failure"; fi | ||
|
|
||
| ) | ||
| __ main "$@" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| #!/usr/bin/env bash | ||
| # Copyright (C) Microsoft Corporation. All rights reserved. | ||
| # Licensed under the MIT License. See LICENSE in project root for information. | ||
|
|
||
| # publish [dirname | --top] | ||
| # Copies the built documentation in $BUILD_ARTIFACTS/docs to the docs container | ||
| # at the specified directory name which defaults to $MML_VERSION; use `--top` to | ||
| # copy it to the toplevel so it's the default docs content. | ||
|
|
||
| . "$(dirname "${BASH_SOURCE[0]}")/../../runme" "$@" | ||
| main() ( | ||
|
|
||
| target="$1"; shift | ||
| if [[ -z "$target" ]]; then target="$MML_VERSION"; fi | ||
| if [[ "x$target" = "x--top" ]]; then | ||
| show section "Publishing docs to toplevel"; target="" | ||
| else | ||
| show section "Publishing docs to $target" | ||
| fi | ||
|
|
||
| default="index.html" | ||
| mime_types=( | ||
| # note: glob patterns are matched against the whole copied path, so they | ||
| # should always start with a `*` | ||
| "*.html;text/html" | ||
| "*.png;image/png" | ||
| "*.gif;image/gif" | ||
| "*.svg;image/svg+xml" | ||
| "*.js;application/javascript" | ||
| "*.css;text/css" | ||
| "*.ttf;application/font-sfnt" | ||
| "*.woff;application/font-woff" | ||
| "*.eot;application/vnd.ms-fontobject" | ||
| "*.txt;text/plain" | ||
| "*.doctree;application/octet-stream" | ||
| "*/.buildinfo;application/octet-stream" | ||
| "*/objects.inv;application/octet-stream" | ||
| "*/environment.pickle;application/octet-stream" | ||
| ) | ||
|
|
||
| cd "$BUILD_ARTIFACTS/docs" | ||
| restore=$(shopt -p dotglob); shopt -s dotglob; files=( ** ); $restore | ||
|
|
||
| # Make $default files when missing | ||
| fst=1 | ||
| for f in . "${files[@]}"; do | ||
| x="$f/$default" | ||
| if [[ ! -d "$f" || -e "$x" ]]; then continue; fi | ||
| if ((fst)); then show - "creating default $default files"; fst=0; fi | ||
| show - " $x" | ||
| { echo "<html><body><pre style=\"font-size: 150%;\">" | ||
| for y in "$f"/*; do | ||
| u="$(basename "$y")"; if [[ -d "$y" ]]; then u+="/"; fi | ||
| if [[ "$u" = "$default" ]]; then continue; fi | ||
| echo "<a href=\"$u\">$u</u>" | ||
| done | ||
| echo "</pre></body></html>" | ||
| } > "$x" | ||
| files+=("$x") | ||
| done | ||
|
|
||
| len=${#files[@]} | ||
|
|
||
| # copy all files with a proper type | ||
| for mt in "${mime_types[@]}"; do | ||
| glob="${mt%;*}"; type="${mt##*;}" | ||
| show - "Copying $glob as $type" | ||
| for ((i=0; i<len; i++)); do | ||
| # mimics the matching that `upload-batch` with `--pattern` is doing | ||
| if [[ -n "${files[$i]}" && "${files[$i]}" = $glob ]]; then unset files[$i]; fi | ||
| done | ||
| collect_log=1 _ azblob upload-batch \ | ||
| --source "." --destination "$DOCS_CONTAINER${target:+/}$target" \ | ||
| --pattern "$glob" --content-type "$type" | ||
| done | ||
|
|
||
| # Deal with directories | ||
| show - "Making directory defaults" | ||
| tmp="/tmp/mmlbuild-$$" | ||
| if [[ -z "$target" ]]; then start=0; else start=-1; fi | ||
| for ((i=-1; i<len; i++)); do | ||
| if ((i<0)) # -1 for the top directory (can't do this for toplevel target) | ||
| then f="."; webdir="$MML_VERSION"; redirect="$MML_VERSION/" | ||
| else f="${files[$i]}"; webdir="$target${target:+/}$f"; redirect="${f##*/}/"; fi | ||
| if [[ -z "$f" || ! -d "$f" ]]; then continue; fi | ||
| unset files[$i] | ||
| if [[ ! -e "$f/$default" ]]; then continue; fi | ||
| # copy "foo/$default" also to "foo/" | ||
| collect_log=1 _ azblob upload \ | ||
| --container "$DOCS_CONTAINER" --content-type "text/html" \ | ||
| --file "$f/$default" --name "$webdir/" \ | ||
| 2> /dev/null | ||
| # create "foo" for redirections to "foo/" | ||
| show command "... html redirect to $redirect ... > $(qstr "$tmp")" | ||
| { echo "<html><head>" | ||
| echo "<meta http-equiv=\"refresh\" content=\"0; url=$redirect\" />" | ||
| echo "</head><body>" | ||
| echo "Moved <a href=\"$redirect\">here</a>" | ||
| echo "</body></html>" | ||
| } > "$tmp" | ||
| collect_log=1 _ azblob upload \ | ||
| --container "$DOCS_CONTAINER" --content-type "text/html" \ | ||
| --file "$tmp" --name "$webdir" \ | ||
| 2> /dev/null | ||
| done | ||
| rm -f "$tmp" | ||
|
|
||
| if ((${#files[@]} > 0)); then | ||
| echo "Leftovers: ${#files[@]}" | ||
| printf ' %s\n' "${files[@]}" | ||
| failwith "the above paths were not in any known patterns" | ||
| fi | ||
|
|
||
| ) | ||
| __ main "$@" |
Oops, something went wrong.