Skip to content

Commit cdba47e

Browse files
committed
Convert "bashbrew" to Go and add a new RFC 2822 manifest file format
To satisfy both Windows (docker-library/golang#92) building and "non-AUFS" (docker-library#1612, docker-library#1537) building ("build machine" constraints, if you will), an update to the manifest file format is necessary. To this end, we're introducing a new format which uses RFC 2822, allowing much more expressivity (especially for additional metadata like `Constraints`) and readability. The new tool also includes backwards compatibility for the older line-based manifest file format, but its usage is discouraged (and we will be slowly converting at least the `docker-library` repos to use the new format in their `generate-stackbrew-library.sh` scripts). One of the tangential benefits of this conversion is a massive increase in the speed of the tool (`bashbrew list --all` on a very fast system with an SSD and a flaming hot cache used to take ~5s and now takes ~0.012s).
1 parent 8b9d1a1 commit cdba47e

File tree

170 files changed

+38108
-599
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

170 files changed

+38108
-599
lines changed

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
11
.git
2+
bashbrew/go/bin
3+
bashbrew/go/pkg
4+
bashbrew/go/vendor/bin
5+
bashbrew/go/vendor/pkg
26
bashbrew/logs
37
bashbrew/src

.travis.yml

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,27 @@
1-
language: bash
1+
language: go
2+
go: 1.6.2
23

34
# allow for use of Docker-based workers
45
sudo: false
56

7+
install:
8+
- |
9+
GB_VERSION=$(awk '$1 == "ENV" && $2 == "GB_VERSION" { print $3; exit }' Dockerfile)
10+
firstGo="${GOPATH%%:*}"
11+
mkdir -p "$firstGo/src/github.com/constabulary"
12+
(
13+
cd "$firstGo/src/github.com/constabulary"
14+
wget -qO- "https://github.com/constabulary/gb/archive/v${GB_VERSION}.tar.gz" \
15+
| tar -xz
16+
mv gb-* gb
17+
cd gb
18+
go install -v ./...
19+
)
20+
621
before_script:
722
- env | sort
23+
- ( cd bashbrew/go && gb build )
24+
- export PATH="$PWD/bashbrew/go/bin:$PATH"
825

9-
# --no-build because we has no Docker in Travis :)
1026
script:
1127
- ./bashbrew/travis.sh

Dockerfile

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,35 @@
11
FROM docker:1.9-git
22

3+
# add "edge" since Alpine 3.3 only has Go 1.5 and we need 1.6+
4+
RUN sed -ri -e 'p; s!^!@edge !; s!v[0-9.]+!edge!' /etc/apk/repositories
5+
36
RUN apk add --update \
47
bash \
8+
go@edge \
59
&& rm -rf /var/cache/apk/*
610

11+
ENV GOPATH /go
12+
ENV PATH $GOPATH/bin:$PATH
13+
14+
ENV GB_VERSION 0.4.1
15+
RUN set -x \
16+
&& mkdir -p /go/src/github.com/constabulary \
17+
&& cd /go/src/github.com/constabulary \
18+
&& wget -qO- "https://github.com/constabulary/gb/archive/v${GB_VERSION}.tar.gz" \
19+
| tar -xz \
20+
&& mv gb-* gb \
21+
&& cd gb \
22+
&& go install -v ./...
23+
724
ENV DIR /usr/src/official-images
25+
ENV PATH $DIR/bashbrew/go/bin:$PATH
26+
27+
ENV BASHBREW_LIBRARY $DIR/library
28+
ENV BASHBREW_CACHE /bashbrew-cache
29+
830
WORKDIR $DIR
931
COPY . $DIR
10-
RUN ln -s "$(readlink -f bashbrew/bashbrew.sh)" /usr/local/bin/bashbrew
1132

12-
VOLUME $DIR/bashbrew/logs $DIR/bashbrew/src
33+
RUN cd bashbrew/go && gb build
34+
35+
VOLUME $BASHBREW_CACHE

README.md

Lines changed: 43 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -201,9 +201,9 @@ The `Dockerfile` should be written to help mitigate man-in-the-middle attacks du
201201

202202
##### Runtime Configuration
203203

204-
By default, Docker containers are executed with reduced privileges: whitelisted Linux capabilities, Control Groups, and a default Seccomp profile (1.10+ w/ host support). Software running in a container may require additional privileges in order to function correctly, and there are a number of command line options to customize container execution. See [`docker run` Reference](https://docs.docker.com/engine/reference/run/) and [Seccomp for Docker](https://docs.docker.com/engine/security/seccomp/) for reference.
204+
By default, Docker containers are executed with reduced privileges: whitelisted Linux capabilities, Control Groups, and a default Seccomp profile (1.10+ w/ host support). Software running in a container may require additional privileges in order to function correctly, and there are a number of command line options to customize container execution. See [`docker run` Reference](https://docs.docker.com/engine/reference/run/) and [Seccomp for Docker](https://docs.docker.com/engine/security/seccomp/) for reference.
205205

206-
Official Repositories that require additional privileges should specify the minimal set of command line options for the software to function, and may still be rejected if this introduces significant portability or security issues. In general, `--privileged` is not allowed, but a combination of `--cap-add` and `--device` options may be acceptable. Additionally, `--volume` can be tricky as there are many host filesystem locations that introduce portability/security issues (i.e. X11 socket).
206+
Official Repositories that require additional privileges should specify the minimal set of command line options for the software to function, and may still be rejected if this introduces significant portability or security issues. In general, `--privileged` is not allowed, but a combination of `--cap-add` and `--device` options may be acceptable. Additionally, `--volume` can be tricky as there are many host filesystem locations that introduce portability/security issues (i.e. X11 socket).
207207

208208
### Commitment
209209

@@ -223,15 +223,52 @@ The filename of a definition file will determine the name of the image repositor
223223

224224
### Instruction format
225225

226-
<docker-tag>: <git-url>@<git-commit-id>
226+
The manifest file format is officially based on [RFC 2822](https://www.ietf.org/rfc/rfc2822.txt), and as such should be familiar to folks who are already familiar with the "headers" of many popular internet protocols/formats such as HTTP or email.
227+
228+
Maintainers: John Smith <jsmith@example.com> (@example-jsmith),
229+
Anne Smith <asmith@example.com> (@example-asmith)
230+
231+
Tags: 4.1.1, 4.1, 4, latest
232+
GitRepo: https://github.com/docker-library/wordpress.git
233+
GitCommit: bbef6075afa043cbfe791b8de185105065c02c01
234+
235+
Tags: 2.6.17, 2.6
236+
GitRepo: https://github.com/docker-library/redis.git
237+
GitCommit: 062335e0a8d20cab2041f25dfff2fbaf58544471
238+
Directory: 2.6
239+
240+
Tags: 13.2, harlequin
241+
GitRepo: https://github.com/openSUSE/docker-containers-build.git
242+
GitFetch: refs/heads/openSUSE-13.1
243+
GitCommit: 0d21bc58cd26da2a0a59588affc506b977d6a846
244+
Directory: docker
245+
Constraints: !aufs
246+
247+
Bashbrew will fetch code out of the Git repository (`GitRepo`) at the commit specified (`GitCommit`). If the commit referenced is not available by fetching `master` of the associated `GitRepo`, it becomes necessary to supply a value for `GitFetch` in order to tell Bashbrew what ref to fetch in order to get the commit necessary.
248+
249+
The built image will be tagged as `<manifest-filename>:<tag>` (ie, `library/golang` with a `Tags` value of `1.6, 1, latest` will create tags of `golang:1.6`, `golang:1`, and `golang:latest`).
250+
251+
Optionally, if `Directory` is present, Bashbrew will look for the `Dockerfile` inside the specified subdirectory instead of at the root (and `Directory` will be used as the ["context" for the build](https://docs.docker.com/reference/builder/) instead of the top-level of the repository).
252+
253+
#### Deprecated format
254+
255+
This is the older, now-deprecated format for library manifest files. Its usage is discouraged (although it is still supported).
256+
257+
# maintainer: Your Name <your@email.com> (@github.name)
258+
259+
# maintainer: John Smith <jsmith@example.com> (@example-jsmith)
260+
# maintainer: Anne Smith <asmith@example.com> (@example-asmith)
261+
262+
263+
<Tag>: <GitRepo>@<GitCommit>
227264

228265
4.1.1: git://github.com/docker-library/wordpress@bbef6075afa043cbfe791b8de185105065c02c01
229266
4.1: git://github.com/docker-library/wordpress@bbef6075afa043cbfe791b8de185105065c02c01
230267
4: git://github.com/docker-library/wordpress@bbef6075afa043cbfe791b8de185105065c02c01
231268
latest: git://github.com/docker-library/wordpress@bbef6075afa043cbfe791b8de185105065c02c01
232269

233270

234-
<docker-tag>: <git-url>@<git-commit-id> <dockerfile-dir>
271+
<Tag>: <GitRepo>@<GitCommit> <Directory>
235272

236273
2.6.17: git://github.com/docker-library/redis@062335e0a8d20cab2041f25dfff2fbaf58544471 2.6
237274
2.6: git://github.com/docker-library/redis@062335e0a8d20cab2041f25dfff2fbaf58544471 2.6
@@ -243,34 +280,24 @@ The filename of a definition file will determine the name of the image repositor
243280

244281
experimental: git://github.com/tianon/dockerfiles@90d86ad63c4a06b7d04d14ad830381b876183b3c debian/experimental
245282

246-
Bashbrew will fetch code out of the Git repository at the commit specified here. The generated image will be tagged as `<manifest-filename>:<docker-tag>`.
247-
248-
Using Git tags instead of explicit Git commit references is supported, but heavily discouraged. For example, if a Git tag is changed on the referenced repository to point to another commit, **the image will not be rebuilt**. Instead, either create a new tag (or reference an exact commit) and submit a pull request.
249-
250-
Optionally, if `<dockerfile-dir>` is present, Bashbrew will look for the `Dockerfile` inside the specified subdirectory instead of at the root (and `<dockerfile-dir>` will be used as the ["context" for the build](https://docs.docker.com/reference/builder/)).
283+
Using Git tags instead of explicit Git commit references is supported for the deprecated format only, but is heavily discouraged. For example, if a Git tag is changed on the referenced repository to point to another commit, **the image will not be rebuilt**. Instead, either create a new tag (or reference an exact commit) and submit a pull request.
251284

252285
### Creating a new repository
253286

254287
- Create a new file in the `library/` folder. Its name will be the name of your repository on the Hub.
255288
- Add your tag definitions using the appropriate syntax (see above).
256-
- Add a line similar to the following to the top of the file:
257-
258-
# maintainer: Your Name <your@email.com> (@github.name)
259-
260289
- Create a pull request adding the file from your forked repository to this one. Please be sure to add details as to what your repository does.
261290

262291
### Adding a new tag in an existing repository (that you're the maintainer of)
263292

264293
- Add your tag definition using the instruction format documented above.
265294
- Create a pull request from your Git repository to this one. Please be sure to add details about what's new, if possible.
266-
- In the pull request comments, feel free to prod the repository's maintainers (found in the relevant `MAINTAINERS` file) using GitHub's @-mentions.
267295

268296
### Change to a tag in an existing repository (that you're the maintainer of)
269297

270298
- Update the relevant tag definition using the instruction format documented above.
271299
- Create a pull request from your Git repository to this one. Please be sure to add details about what's changed, if possible.
272-
- In the pull request comments, feel free to prod the repository's maintainers (found in the relevant `MAINTAINERS` file) using GitHub's @-mentions.
273300

274301
## Bashbrew
275302

276-
Bashbrew is a set of bash scripts for cloning, building, tagging, and pushing the Docker official images. See [`README.md` in the `bashbrew/` subfolder](bashbrew/README.md) for more information.
303+
Bashbrew (`bashbrew`) is a tool for cloning, building, tagging, and pushing the Docker official images. See [`README.md` in the `bashbrew/` subfolder](bashbrew/README.md) for more information.

bashbrew/README.md

Lines changed: 59 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -1,120 +1,81 @@
1-
# Bashbrew
2-
3-
The recommended way to use `bashbrew.sh` is to install a symlink in your `PATH` somewhere as `bashbrew`, for example `~/bin/bashbrew -> /path/to/official-images/bashbrew/bashbrew.sh` (assuming `~/bin` is in `PATH`).
1+
# Bashbrew (`bashbrew`)
42

53
```console
64
$ bashbrew --help
7-
8-
usage: bashbrew [build|push|list] [options] [repo[:tag] ...]
9-
ie: bashbrew build --all
10-
bashbrew push debian ubuntu:12.04
11-
bashbrew list --namespaces='_' debian:7 hello-world
12-
13-
This script processes the specified Docker images using the corresponding
14-
repository manifest files.
15-
16-
common options:
17-
--all Build all repositories specified in library
18-
--docker="docker"
19-
Use a custom Docker binary
20-
--retries="4"
21-
How many times to try again if the build/push fails before
22-
considering it a lost cause (always attempts a minimum of
23-
one time, but maximum of one plus this number)
24-
--help, -h, -? Print this help message
25-
--library="/home/tianon/docker/stackbrew/library"
26-
Where to find repository manifest files
27-
--logs="/home/tianon/docker/stackbrew/bashbrew/logs"
28-
Where to store the build logs
29-
--namespaces="_"
30-
Space separated list of image namespaces to act upon
31-
32-
Note that "_" is a special case here for the unprefixed
33-
namespace (ie, "debian" vs "library/debian"), and as such
34-
will be implicitly ignored by the "push" subcommand
35-
36-
Also note that "build" will always tag to the unprefixed
37-
namespace because it is necessary to do so for dependent
38-
images to use FROM correctly (think "onbuild" variants that
39-
are "FROM base-image:some-version")
40-
--uniq
41-
Only process the first tag of identical images
42-
This is not recommended for build or push
43-
i.e. process python:2.7, but not python:2
44-
45-
build options:
46-
--no-build Don't build, print what would build
47-
--no-clone Don't pull/clone Git repositories
48-
--src="/home/tianon/docker/stackbrew/bashbrew/src"
49-
Where to store cloned Git repositories (GOPATH style)
50-
51-
push options:
52-
--no-push Don't push, print what would push
5+
NAME:
6+
bashbrew - canonical build tool for the official images
7+
8+
USAGE:
9+
bashbrew [global options] command [command options] [arguments...]
10+
11+
COMMANDS:
12+
list list repo:tag combinations for a given repo
13+
build build (and tag) repo:tag combinations for a given repo
14+
tag tag repo:tag into a namespace (especially for pushing)
15+
push push namespace/repo:tag (see also "tag")
16+
17+
plumbing:
18+
children print the repos built FROM a given repo or repo:tag
19+
parents print the repos this repo or repo:tag is FROM
20+
cat print manifest contents for repo or repo:tag
21+
from print FROM for repo:tag
22+
23+
GLOBAL OPTIONS:
24+
--verbose, -v enable more output (esp. "docker build" output) [$BASHBREW_VERBOSE]
25+
--no-sort do not apply any sorting, even via --build-order
26+
--constraint value build constraints (see Constraints in Manifest2822Entry)
27+
--exclusive-constraints skip entries which do not have Constraints
28+
--library value where the bodies are buried (default: "/home/jsmith/docker/official-images/library") [$BASHBREW_LIBRARY]
29+
--cache value where the git wizardry is stashed (default: "/home/jsmith/.cache/bashbrew") [$BASHBREW_CACHE]
30+
--help, -h, -? show help
5331

5432
```
5533

56-
## Subcommands
57-
58-
### `bashbrew build`
34+
## Building
5935

60-
This script reads the library files for the images specified and then clones the required Git repositories into the specified `--src` directory. If the Git repository already exists, the script verifies that the Git ref specified in the library file exists and does `git fetch` as necessary.
36+
Bashbrew itself is built using `gb` ([github.com/constabulary/gb](https://github.com/constabulary/gb)).
6137

62-
The next step in the script is to build each image specified. All the `image:tag` combinations are placed into a queue. The processing order is determined by the order of items passed in on the command line (or alphabetical if `--all` is used). When a whole image, like `debian`, is specified the `image:tag` combinations are added to the queue in the order that they appear in the library file. For each `image:tag` to be processed, the system checks out the specified commit and sets mtimes (see [`git-set-mtimes`](#git-set-mtimes)) of all files in the Git repository to take advantage of Docker caching. If the `image:tag` is `FROM` another image that is later in the queue, it is deferred to the end of the queue.
38+
Once in the `go` subdirectory, `gb build` should produce `go/bin/bashbrew`, ready for use.
6339

64-
After the image is built, the final step of the script is to tag the image into each of the given `--namespaces`.
40+
## Usage
6541

66-
The `--no-clone` option skips the `git clone` step and will cause the script to fail on the build step if the Git repository does not exist or is missing the required Git refs.
42+
In general, `bashbrew build some-repo` or `bashbrew build ./some-file` should be sufficient for using the tool at a surface level, especially for testing. For more complex usage, please see the built-in help (`bashbrew --help`, `bashbrew build --help`, etc).
6743

68-
The `--no-build` option skips all the building, including setting the mtimes.
44+
## Configuration
6945

70-
**WARNING:** setting `--src` so that it uses a local working copy of your Git directory for a specified build will delete untracked and uncommitted changes, and will disable `gc.auto`. It is not recommended to symlink in your working directories for use during build.
46+
The default "flags" configuration is in `~/.config/bashbrew/flags`, but the base path can be overridden with `--config` or `BASHBREW_CONFIG` (technically, the full path to the `flags` configuration file is `${BASHBREW_CONFIG:-${XDG_CONFIG_HOME:-$HOME/.config}/bashbrew}/flags`).
7147

72-
### `bashbrew push`
48+
To set a default namespace for specific commands only:
7349

74-
This script takes advantage of `docker login` and does a `docker push` on each `image:tag` specified for the given `--namespaces`. The script will warn if a given `namespace/image:tag` does not exist.
75-
76-
The `--no-push` option prints out the `docker push` instructions that would have been executed.
50+
```
51+
Commands: tag, push
52+
Namespace: officialstaging
53+
```
7754

78-
### `bashbrew list`
55+
To set a default namespace for all commands:
7956

80-
Takes the same arguments as `bashbrew build` and `bashbrew push`, but prints a list of image names and quits.
57+
```
58+
Namespace: jsmith
59+
```
8160

82-
For example:
61+
A more complex example:
8362

84-
```console
85-
$ # count the number of tags in the official library
86-
$ bashbrew list --all | wc -l
87-
802
88-
$ # count the number of _unique_ tags in the official library
89-
$ bashbrew list --all --uniq | wc -l
90-
296
91-
92-
$ # pull all officially supported tags of "debian"
93-
$ bashbrew list debian | xargs -n1 --verbose docker pull
94-
...
95-
96-
$ # list all unique supported tags of "python"
97-
$ bashbrew list --uniq python
98-
python:2.7.10
99-
python:2.7.10-onbuild
100-
python:2.7.10-slim
101-
python:2.7.10-wheezy
102-
python:3.2.6
103-
python:3.2.6-onbuild
104-
python:3.2.6-slim
105-
python:3.2.6-wheezy
106-
python:3.3.6
107-
python:3.3.6-onbuild
108-
python:3.3.6-slim
109-
python:3.3.6-wheezy
110-
python:3.4.3
111-
python:3.4.3-onbuild
112-
python:3.4.3-slim
113-
python:3.4.3-wheezy
11463
```
64+
# comments are allowed anywhere (and are ignored)
65+
Library: /mnt/docker/official-images/library
66+
Cache: /mnt/docker/bashbrew-cache
67+
Constraints: aufs, docker-1.9, tianon
68+
ExclusiveConstraints: true
11569
116-
## Helper Scripts
70+
# namespace just "tag" and "push" (not "build")
71+
Commands: tag, push
72+
Namespace: tianon
11773
118-
### `git-set-mtimes`
74+
Commands: list
75+
ApplyConstraints: true
76+
77+
Commands: tag
78+
Verbose: true
79+
```
11980

120-
Since Docker caching of layers is based upon the mtimes of files and folders, this script sets each file's mtime to the time of the commit that most recently modified it and sets each directory's mtime to be the most recent mtime of any file or folder contained within it. This gives a deterministic time for all files and folders in the Git repository.
81+
In this example, `bashbrew tag` will get both `Namespace` and `Verbose` applied.

0 commit comments

Comments
 (0)