Skip to content

Commit a7aa9af

Browse files
consideRatioconsideRatio
committed
Provide frozen requirements.txt files for images, and automation
1 parent 329a82d commit a7aa9af

File tree

8 files changed

+262
-19
lines changed

8 files changed

+262
-19
lines changed

‎.github/workflows/refreeze-dockerfile-requirements-txt.yaml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# This is a GitHub workflow defining a set of jobs with a set of steps.
2+
# ref: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions
3+
#
4+
name: Refreeze Dockerfile.requirements.txt
5+
6+
on:
7+
push:
8+
paths:
9+
- "**/Dockerfile"
10+
- "**/Dockerfile.requirements.in"
11+
- "**/Dockerfile.requirements.txt"
12+
- ".github/workflows/refreeze-dockerfile-requirements-txt.yaml"
13+
branches: ["main"]
14+
tags: ["**"]
15+
workflow_dispatch:
16+
17+
jobs:
18+
refreeze-dockerfile-requirements-txt:
19+
name: Refreeze Dockerfile.requirements.txt
20+
21+
# Don't run this job on forks
22+
if: github.repository == 'dask/dask-gateway'
23+
runs-on: ubuntu-latest
24+
25+
strategy:
26+
fail-fast: false
27+
matrix:
28+
include:
29+
- image: dask-gateway
30+
- image: dask-gateway-server
31+
32+
steps:
33+
- uses: actions/checkout@v3
34+
35+
- name: Refreeze Dockerfile.requirements.txt based on Dockerfile.requirements.in
36+
run: |
37+
cd ${{ matrix.image }}
38+
docker run --rm \
39+
--env=CUSTOM_COMPILE_COMMAND='Use "Run workflow" button at https://github.com/dask/dask-gateway/actions/workflows/refreeze-dockerfile-requirements-txt.yaml' \
40+
--volume=$PWD:/opt/${{ matrix.image }} \
41+
--workdir=/opt/${{ matrix.image }} \
42+
--user=root \
43+
python:3.10-slim-bullseye \
44+
sh -c 'pip install pip-tools==6.* && pip-compile --upgrade --output-file=Dockerfile.requirements.txt Dockerfile.requirements.in'
45+
46+
- name: git diff
47+
run: git --no-pager diff --color=always
48+
49+
# ref: https://github.com/peter-evans/create-pull-request
50+
- name: Create a PR
51+
uses: peter-evans/create-pull-request@v4
52+
with:
53+
token: "${{ secrets.dask_bot_token }}"
54+
author: Dask Bot Account <65357765+dask-bot@users.noreply.github.com>
55+
committer: Dask Bot Account <65357765+dask-bot@users.noreply.github.com>
56+
branch: update-image-${{ matrix.image }}
57+
labels: dependencies
58+
commit-message: Refreeze ${{ matrix.image}}/Dockerfile.requirements.txt
59+
title: Refreeze ${{ matrix.image}}/Dockerfile.requirements.txt
60+
body: >-
61+
${{ matrix.image}}/Dockerfile.requirements.txt has been refrozen
62+
based on ${{ matrix.image}}/Dockerfile.requirements.in.

‎RELEASE.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,11 @@
22

33
Follow these steps to cut a new release.
44

5+
- Update frozen Dockerfile.requirements.txt files by running workflow that has
6+
automated this at
7+
https://github.com/dask/dask-gateway/actions/workflows/refreeze-dockerfile-requirements-txt.yaml
8+
and merging the PRs.
9+
510
- Update `docs/changelog.md` in a dedicated PR
611

712
- Generate a list of PRs using [executablebooks/github-activity](https://github.com/executablebooks/github-activity)

‎dask-gateway-server/Dockerfile‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,9 @@ WORKDIR /home/dask/
3333
# The Golang proxy binary isn't built as the dask-gateway Helm chart relies on
3434
# Traefik as a proxy instead to run in its dedicated pod.
3535
#
36-
COPY --chown=dask:dask . /srv/dask-gateway-server
37-
36+
COPY --chown=dask:dask . /opt/dask-gateway-server
3837
RUN DASK_GATEWAY_SERVER__NO_PROXY=true pip install --no-cache-dir \
39-
"/srv/dask-gateway-server[kubernetes]"
38+
-r /opt/dask-gateway-server/Dockerfile.requirements.txt
4039

4140
ENTRYPOINT ["tini", "-g", "--"]
4241
CMD ["dask-gateway-server", "--config", "/etc/dask-gateway/dask_gateway_config.py"]

‎dask-gateway-server/Dockerfile.requirements.in‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# These are the requirements we know we want to install in the Dockerfile, and
2+
# then we freeze them ahead of time to provide a clear description of the
3+
# dependencies we have installed.
4+
#
5+
6+
# Install dask-gateway-server[kubernetes], which is the only thing needed for
7+
# our CI test suite.
8+
#
9+
.[kubernetes]

‎dask-gateway-server/Dockerfile.requirements.txt‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#
2+
# This file is autogenerated by pip-compile with python 3.10
3+
# To update, run:
4+
#
5+
# Use "Run workflow" button at https://github.com/dask/dask-gateway/actions/workflows/refreeze-dockerfile-requirements-txt.yaml
6+
#
7+
aiohttp==3.8.1
8+
# via
9+
# dask-gateway-server
10+
# kubernetes-asyncio
11+
aiosignal==1.2.0
12+
# via aiohttp
13+
async-timeout==4.0.2
14+
# via aiohttp
15+
attrs==21.4.0
16+
# via aiohttp
17+
certifi==2022.5.18.1
18+
# via kubernetes-asyncio
19+
cffi==1.15.0
20+
# via cryptography
21+
charset-normalizer==2.0.12
22+
# via aiohttp
23+
colorlog==6.6.0
24+
# via dask-gateway-server
25+
cryptography==37.0.2
26+
# via dask-gateway-server
27+
dask-gateway-server @ file:///opt/dask-gateway-server
28+
# via -r Dockerfile.requirements.in
29+
frozenlist==1.3.0
30+
# via
31+
# aiohttp
32+
# aiosignal
33+
idna==3.3
34+
# via yarl
35+
kubernetes-asyncio==22.6.5
36+
# via dask-gateway-server
37+
multidict==6.0.2
38+
# via
39+
# aiohttp
40+
# yarl
41+
pycparser==2.21
42+
# via cffi
43+
python-dateutil==2.8.2
44+
# via kubernetes-asyncio
45+
pyyaml==6.0
46+
# via kubernetes-asyncio
47+
six==1.16.0
48+
# via
49+
# kubernetes-asyncio
50+
# python-dateutil
51+
traitlets==5.2.2.post1
52+
# via dask-gateway-server
53+
urllib3==1.26.9
54+
# via kubernetes-asyncio
55+
yarl==1.7.2
56+
# via aiohttp
57+
58+
# The following packages are considered to be unsafe in a requirements file:
59+
# setuptools

‎dask-gateway/Dockerfile‎

Lines changed: 2 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -34,22 +34,8 @@ WORKDIR /home/dask/
3434

3535
# Install dask-gateway
3636
COPY --chown=dask:dask . /opt/dask-gateway
37-
# Install dask-gateway, which is the only thing needed for our CI test suite.
38-
#
39-
# We also install the bare minimum to provide end users with a primitive
40-
# end-to-end demonstrative test doing work in the worker pods and accessing the
41-
# scheduler dashboard without changing the image.
42-
#
43-
# - bokeh is required by the scheduler pod to present dashbaords.
44-
# - numpy is required for running a basic computation test:
45-
# https://gateway.dask.org/usage.html#run-computations-on-the-cluster
46-
#
47-
COPY --chown=dask:dask . /opt/dask-gateway
48-
RUN pip install --no-cache-dir
49-
/opt/dask-gateway \
50-
&& pip install --no-cache-dir \
51-
bokeh \
52-
numpy
37+
RUN pip install --no-cache-dir \
38+
-r /opt/dask-gateway/Dockerfile.requirements.txt
5339

5440
# Only set ENTRYPOINT, CMD is configured at runtime by dask-gateway-server
5541
ENTRYPOINT ["tini", "-g", "--"]

‎dask-gateway/Dockerfile.requirements.in‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# These are the requirements we know we want to install in the Dockerfile, and
2+
# then we freeze them ahead of time to provide a clear description of the
3+
# dependencies we have installed.
4+
#
5+
6+
# Install dask-gateway, which is the only thing needed for our CI test suite.
7+
.
8+
9+
# We also install the bare minimum to provide end users with a primitive
10+
# end-to-end demonstrative test doing work in the worker pods and accessing the
11+
# scheduler dashboard without changing the image.
12+
#
13+
# - bokeh is required by the scheduler pod to present dashbaords.
14+
# - numpy is required for running a basic computation test:
15+
# https://gateway.dask.org/usage.html#run-computations-on-the-cluster
16+
#
17+
bokeh
18+
numpy

‎dask-gateway/Dockerfile.requirements.txt‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#
2+
# This file is autogenerated by pip-compile with python 3.10
3+
# To update, run:
4+
#
5+
# Use "Run workflow" button at https://github.com/dask/dask-gateway/actions/workflows/refreeze-dockerfile-requirements-txt.yaml
6+
#
7+
aiohttp==3.8.1
8+
# via dask-gateway
9+
aiosignal==1.2.0
10+
# via aiohttp
11+
async-timeout==4.0.2
12+
# via aiohttp
13+
attrs==21.4.0
14+
# via aiohttp
15+
bokeh==2.4.3
16+
# via -r Dockerfile.requirements.in
17+
charset-normalizer==2.0.12
18+
# via aiohttp
19+
click==8.1.3
20+
# via
21+
# dask-gateway
22+
# distributed
23+
cloudpickle==2.1.0
24+
# via
25+
# dask
26+
# distributed
27+
dask==2022.6.0
28+
# via
29+
# dask-gateway
30+
# distributed
31+
dask-gateway @ file:///opt/dask-gateway
32+
# via -r Dockerfile.requirements.in
33+
distributed==2022.6.0
34+
# via dask-gateway
35+
frozenlist==1.3.0
36+
# via
37+
# aiohttp
38+
# aiosignal
39+
fsspec==2022.5.0
40+
# via dask
41+
heapdict==1.0.1
42+
# via zict
43+
idna==3.3
44+
# via yarl
45+
jinja2==3.1.2
46+
# via
47+
# bokeh
48+
# distributed
49+
locket==1.0.0
50+
# via
51+
# distributed
52+
# partd
53+
markupsafe==2.1.1
54+
# via jinja2
55+
msgpack==1.0.4
56+
# via distributed
57+
multidict==6.0.2
58+
# via
59+
# aiohttp
60+
# yarl
61+
numpy==1.22.4
62+
# via
63+
# -r Dockerfile.requirements.in
64+
# bokeh
65+
packaging==21.3
66+
# via
67+
# bokeh
68+
# dask
69+
# distributed
70+
partd==1.2.0
71+
# via dask
72+
pillow==9.1.1
73+
# via bokeh
74+
psutil==5.9.1
75+
# via distributed
76+
pyparsing==3.0.9
77+
# via packaging
78+
pyyaml==6.0
79+
# via
80+
# bokeh
81+
# dask
82+
# dask-gateway
83+
# distributed
84+
sortedcontainers==2.4.0
85+
# via distributed
86+
tblib==1.7.0
87+
# via distributed
88+
toolz==0.11.2
89+
# via
90+
# dask
91+
# distributed
92+
# partd
93+
tornado==6.1
94+
# via
95+
# bokeh
96+
# dask-gateway
97+
# distributed
98+
typing-extensions==4.2.0
99+
# via bokeh
100+
urllib3==1.26.9
101+
# via distributed
102+
yarl==1.7.2
103+
# via aiohttp
104+
zict==2.2.0
105+
# via distributed

0 commit comments

Comments
 (0)