Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/docker-wiki-update.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: Download build manifests from GitHub artifacts and push them to GitHub wik
# This way we make sure we don't access wiki pages from several jobs simultaneously

env:
PUSH_TO_REGISTRY: ${{ (github.repository_owner == 'jupyter' || github.repository_owner == 'mathbunnyru') && (github.ref == 'refs/heads/main' || github.event_name == 'schedule') }}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think wiki is created automatically, and given these changes we can actually generate wiki changes in forks as well

PUSH_TO_REGISTRY: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}

on:
workflow_call:
Expand Down Expand Up @@ -44,6 +44,7 @@ jobs:
--wiki-dir wiki/
--hist-lines-dir /tmp/jupyter/hist_lines/
--manifests-dir /tmp/jupyter/manifests/
--repository ${{ github.repository }}
shell: bash

- name: Push Wiki to GitHub 📤
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -463,7 +463,7 @@ jobs:
- image: pytorch-notebook
variant: cuda12
needs: [aarch64-images-tag-push, x86_64-images-tag-push]
if: (github.repository_owner == 'jupyter' || github.repository_owner == 'mathbunnyru') && !contains(github.event.pull_request.title, '[FAST_BUILD]')
if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}

merge-tags-fast:
uses: ./.github/workflows/docker-merge-tags.yml
Expand All @@ -478,19 +478,19 @@ jobs:
image: [docker-stacks-foundation, base-notebook]
variant: [default]
needs: [aarch64-images-tag-push-fast, x86_64-images-tag-push-fast]
if: (github.repository_owner == 'jupyter' || github.repository_owner == 'mathbunnyru') && contains(github.event.pull_request.title, '[FAST_BUILD]')
if: contains(github.event.pull_request.title, '[FAST_BUILD]')

wiki-update:
uses: ./.github/workflows/docker-wiki-update.yml
needs: [aarch64-images-tag-push, x86_64-images-tag-push]
if: github.repository_owner == 'jupyter' && !contains(github.event.pull_request.title, '[FAST_BUILD]')
if: ${{ !contains(github.event.pull_request.title, '[FAST_BUILD]') }}
permissions:
contents: write

wiki-update-fast:
uses: ./.github/workflows/docker-wiki-update.yml
needs: [aarch64-images-tag-push-fast, x86_64-images-tag-push-fast]
if: github.repository_owner == 'jupyter' && contains(github.event.pull_request.title, '[FAST_BUILD]')
if: contains(github.event.pull_request.title, '[FAST_BUILD]')
permissions:
contents: write

Expand Down
27 changes: 27 additions & 0 deletions tagging/Home.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Jupyter Docker Stacks build manifests
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding this file to a repo makes it easier to make PRs - now we have even less reasons to manipulate wiki repo manually.
And forks will get nice wiki pages by default


<!-- Note: this file is copied to wiki from the main repo, edits on wiki page will be overridden -->
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this comment to show that edits on the wiki page are not saved


Welcome!
Please see [the documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/) for help with
using, contributing to, and maintaining the Jupyter Docker stacks images.

## Build History

This is an auto-generated index of information from the build system.
In this index, you can find image tags, links to commits, and build manifests that describe the image.
All the builds are grouped by year and then month.

Note: we only store the last 4500 manifest files because of GitHub limits.
That's why old manifest files might not be available.
If you want to clone this repo and access the Git history, use the following command: `git clone git@github.com:{REPOSITORY}.wiki.git`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In forks this will be a link to a fork wiki


In the tables below, each line represents:

- `YYYY-MM`: link to a page with a list of images built
- `Builds`: # of times build workflow finished
- `Images`: # of single platform images pushed
- `Commits`: # of commits made and a GitHub link

<!-- Everything below is auto-generated, all manual changes will be erased -->
<!-- YEAR_MONTHLY_TABLES -->
19 changes: 13 additions & 6 deletions tagging/update_wiki.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
git = plumbum.local["git"]

LOGGER = logging.getLogger(__name__)
THIS_DIR = Path(__file__).parent.resolve()


def calculate_monthly_stat(
Expand Down Expand Up @@ -46,16 +47,16 @@ def calculate_monthly_stat(
return builds, images, commits


def regenerate_home_wiki_page(wiki_dir: Path) -> None:
def generate_home_wiki_page(wiki_dir: Path, repository: str) -> None:
YEAR_MONTHLY_TABLES = "<!-- YEAR_MONTHLY_TABLES -->\n"

wiki_home_file = wiki_dir / "Home.md"
wiki_home_content = wiki_home_file.read_text()
wiki_home_content = (THIS_DIR / "Home.md").read_text()

assert YEAR_MONTHLY_TABLES in wiki_home_content
wiki_home_content = wiki_home_content[
: wiki_home_content.find(YEAR_MONTHLY_TABLES) + len(YEAR_MONTHLY_TABLES)
]
wiki_home_content = wiki_home_content.format(REPOSITORY=repository)

YEAR_TABLE_HEADER = """\
## {year}
Expand All @@ -65,7 +66,7 @@ def regenerate_home_wiki_page(wiki_dir: Path) -> None:
"""

GITHUB_COMMITS_URL = (
"[{}](https://github.com/jupyter/docker-stacks/commits/main/?since={}&until={})"
f"[{{}}](https://github.com/{repository}/commits/main/?since={{}}&until={{}})"
)

for year_dir in sorted((wiki_dir / "monthly-files").glob("*"), reverse=True):
Expand Down Expand Up @@ -95,7 +96,7 @@ def regenerate_home_wiki_page(wiki_dir: Path) -> None:
year_total_line = f"| **Total** | {year_builds: <6} | {year_images: <6} | {year_commits_url: <95} |\n"
wiki_home_content += year_total_line

wiki_home_file.write_text(wiki_home_content)
(wiki_dir / "Home.md").write_text(wiki_home_content)
LOGGER.info("Updated Home page")


Expand Down Expand Up @@ -159,6 +160,7 @@ def update_wiki(
wiki_dir: Path,
hist_lines_dir: Path,
manifests_dir: Path,
repository: str,
allow_no_files: bool,
) -> None:
LOGGER.info("Updating wiki")
Expand All @@ -185,7 +187,7 @@ def update_wiki(
year_month = build_history_line[3:10]
update_monthly_wiki_page(wiki_dir, year_month, build_history_line)

regenerate_home_wiki_page(wiki_dir)
generate_home_wiki_page(wiki_dir, repository)
remove_old_manifests(wiki_dir)


Expand All @@ -211,6 +213,11 @@ def update_wiki(
type=Path,
help="Directory with manifest files",
)
arg_parser.add_argument(
"--repository",
required=True,
help="Repository name on GitHub",
)
arg_parser.add_argument(
"--allow-no-files",
action="store_true",
Expand Down