-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Convert "bashbrew" to Go and add a new RFC 2822 manifest file format
To satisfy both Windows (docker-library/golang#92) building and "non-AUFS" (#1612, #1537) building ("build machine" constraints, if you will), an update to the manifest file format is necessary. To this end, we're introducing a new format which uses RFC 2822, allowing much more expressivity (especially for additional metadata like `Constraints`) and readability. The new tool also includes backwards compatibility for the older line-based manifest file format, but its usage is discouraged (and we will be slowly converting at least the `docker-library` repos to use the new format in their `generate-stackbrew-library.sh` scripts). One of the tangential benefits of this conversion is a massive increase in the speed of the tool (`bashbrew list --all` on a very fast system with an SSD and a flaming hot cache used to take ~5s and now takes ~0.012s).
- Loading branch information
Showing
171 changed files
with
38,157 additions
and
610 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,7 @@ | ||
| .git | ||
| bashbrew/go/bin | ||
| bashbrew/go/pkg | ||
| bashbrew/go/vendor/bin | ||
| bashbrew/go/vendor/pkg | ||
| bashbrew/logs | ||
| bashbrew/src |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,27 @@ | ||
| language: bash | ||
| language: go | ||
| go: 1.6.2 | ||
|
|
||
| # allow for use of Docker-based workers | ||
| sudo: false | ||
|
|
||
| install: | ||
| - | | ||
| GB_VERSION=$(awk '$1 == "ENV" && $2 == "GB_VERSION" { print $3; exit }' Dockerfile) | ||
| firstGo="${GOPATH%%:*}" | ||
| mkdir -p "$firstGo/src/github.com/constabulary" | ||
| ( | ||
| cd "$firstGo/src/github.com/constabulary" | ||
| wget -qO- "https://github.com/constabulary/gb/archive/v${GB_VERSION}.tar.gz" \ | ||
| | tar -xz | ||
| mv gb-* gb | ||
| cd gb | ||
| go install -v ./... | ||
| ) | ||
| before_script: | ||
| - env | sort | ||
| - ( cd bashbrew/go && gb build ) | ||
| - export PATH="$PWD/bashbrew/go/bin:$PATH" | ||
|
|
||
| # --no-build because we has no Docker in Travis :) | ||
| script: | ||
| - ./bashbrew/travis.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,35 @@ | ||
| FROM docker:1.9-git | ||
|
|
||
| # add "edge" since Alpine 3.3 only has Go 1.5 and we need 1.6+ | ||
| RUN sed -ri -e 'p; s!^!@edge !; s!v[0-9.]+!edge!' /etc/apk/repositories | ||
|
|
||
| RUN apk add --update \ | ||
| bash \ | ||
| go@edge \ | ||
| && rm -rf /var/cache/apk/* | ||
|
|
||
| ENV GOPATH /go | ||
| ENV PATH $GOPATH/bin:$PATH | ||
|
|
||
| ENV GB_VERSION 0.4.1 | ||
| RUN set -x \ | ||
| && mkdir -p /go/src/github.com/constabulary \ | ||
| && cd /go/src/github.com/constabulary \ | ||
| && wget -qO- "https://github.com/constabulary/gb/archive/v${GB_VERSION}.tar.gz" \ | ||
| | tar -xz \ | ||
| && mv gb-* gb \ | ||
| && cd gb \ | ||
| && go install -v ./... | ||
|
|
||
| ENV DIR /usr/src/official-images | ||
| ENV PATH $DIR/bashbrew/go/bin:$PATH | ||
|
|
||
| ENV BASHBREW_LIBRARY $DIR/library | ||
| ENV BASHBREW_CACHE /bashbrew-cache | ||
|
|
||
| WORKDIR $DIR | ||
| COPY . $DIR | ||
| RUN ln -s "$(readlink -f bashbrew/bashbrew.sh)" /usr/local/bin/bashbrew | ||
|
|
||
| VOLUME $DIR/bashbrew/logs $DIR/bashbrew/src | ||
| RUN cd bashbrew/go && gb build | ||
|
|
||
| VOLUME $BASHBREW_CACHE |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,120 +1,81 @@ | ||
| # Bashbrew | ||
|
|
||
| The recommended way to use `bashbrew.sh` is to install a symlink in your `PATH` somewhere as `bashbrew`, for example `~/bin/bashbrew -> /path/to/official-images/bashbrew/bashbrew.sh` (assuming `~/bin` is in `PATH`). | ||
| # Bashbrew (`bashbrew`) | ||
|
|
||
| ```console | ||
| $ bashbrew --help | ||
|
|
||
| usage: bashbrew [build|push|list] [options] [repo[:tag] ...] | ||
| ie: bashbrew build --all | ||
| bashbrew push debian ubuntu:12.04 | ||
| bashbrew list --namespaces='_' debian:7 hello-world | ||
|
|
||
| This script processes the specified Docker images using the corresponding | ||
| repository manifest files. | ||
|
|
||
| common options: | ||
| --all Build all repositories specified in library | ||
| --docker="docker" | ||
| Use a custom Docker binary | ||
| --retries="4" | ||
| How many times to try again if the build/push fails before | ||
| considering it a lost cause (always attempts a minimum of | ||
| one time, but maximum of one plus this number) | ||
| --help, -h, -? Print this help message | ||
| --library="/home/tianon/docker/stackbrew/library" | ||
| Where to find repository manifest files | ||
| --logs="/home/tianon/docker/stackbrew/bashbrew/logs" | ||
| Where to store the build logs | ||
| --namespaces="_" | ||
| Space separated list of image namespaces to act upon | ||
| Note that "_" is a special case here for the unprefixed | ||
| namespace (ie, "debian" vs "library/debian"), and as such | ||
| will be implicitly ignored by the "push" subcommand | ||
| Also note that "build" will always tag to the unprefixed | ||
| namespace because it is necessary to do so for dependent | ||
| images to use FROM correctly (think "onbuild" variants that | ||
| are "FROM base-image:some-version") | ||
| --uniq | ||
| Only process the first tag of identical images | ||
| This is not recommended for build or push | ||
| i.e. process python:2.7, but not python:2 | ||
|
|
||
| build options: | ||
| --no-build Don't build, print what would build | ||
| --no-clone Don't pull/clone Git repositories | ||
| --src="/home/tianon/docker/stackbrew/bashbrew/src" | ||
| Where to store cloned Git repositories (GOPATH style) | ||
|
|
||
| push options: | ||
| --no-push Don't push, print what would push | ||
| NAME: | ||
| bashbrew - canonical build tool for the official images | ||
|
|
||
| USAGE: | ||
| bashbrew [global options] command [command options] [arguments...] | ||
|
|
||
| COMMANDS: | ||
| list list repo:tag combinations for a given repo | ||
| build build (and tag) repo:tag combinations for a given repo | ||
| tag tag repo:tag into a namespace (especially for pushing) | ||
| push push namespace/repo:tag (see also "tag") | ||
|
|
||
| plumbing: | ||
| children print the repos built FROM a given repo or repo:tag | ||
| parents print the repos this repo or repo:tag is FROM | ||
| cat print manifest contents for repo or repo:tag | ||
| from print FROM for repo:tag | ||
|
|
||
| GLOBAL OPTIONS: | ||
| --verbose, -v enable more output (esp. "docker build" output) [$BASHBREW_VERBOSE] | ||
| --no-sort do not apply any sorting, even via --build-order | ||
| --constraint value build constraints (see Constraints in Manifest2822Entry) | ||
| --exclusive-constraints skip entries which do not have Constraints | ||
| --library value where the bodies are buried (default: "/home/jsmith/docker/official-images/library") [$BASHBREW_LIBRARY] | ||
| --cache value where the git wizardry is stashed (default: "/home/jsmith/.cache/bashbrew") [$BASHBREW_CACHE] | ||
| --help, -h, -? show help | ||
|
|
||
| ``` | ||
|
|
||
| ## Subcommands | ||
|
|
||
| ### `bashbrew build` | ||
| ## Building | ||
|
|
||
| This script reads the library files for the images specified and then clones the required Git repositories into the specified `--src` directory. If the Git repository already exists, the script verifies that the Git ref specified in the library file exists and does `git fetch` as necessary. | ||
| Bashbrew itself is built using `gb` ([github.com/constabulary/gb](https://github.com/constabulary/gb)). | ||
|
|
||
| The next step in the script is to build each image specified. All the `image:tag` combinations are placed into a queue. The processing order is determined by the order of items passed in on the command line (or alphabetical if `--all` is used). When a whole image, like `debian`, is specified the `image:tag` combinations are added to the queue in the order that they appear in the library file. For each `image:tag` to be processed, the system checks out the specified commit and sets mtimes (see [`git-set-mtimes`](#git-set-mtimes)) of all files in the Git repository to take advantage of Docker caching. If the `image:tag` is `FROM` another image that is later in the queue, it is deferred to the end of the queue. | ||
| Once in the `go` subdirectory, `gb build` should produce `go/bin/bashbrew`, ready for use. | ||
|
|
||
| After the image is built, the final step of the script is to tag the image into each of the given `--namespaces`. | ||
| ## Usage | ||
|
|
||
| The `--no-clone` option skips the `git clone` step and will cause the script to fail on the build step if the Git repository does not exist or is missing the required Git refs. | ||
| In general, `bashbrew build some-repo` or `bashbrew build ./some-file` should be sufficient for using the tool at a surface level, especially for testing. For more complex usage, please see the built-in help (`bashbrew --help`, `bashbrew build --help`, etc). | ||
|
|
||
| The `--no-build` option skips all the building, including setting the mtimes. | ||
| ## Configuration | ||
|
|
||
| **WARNING:** setting `--src` so that it uses a local working copy of your Git directory for a specified build will delete untracked and uncommitted changes, and will disable `gc.auto`. It is not recommended to symlink in your working directories for use during build. | ||
| The default "flags" configuration is in `~/.config/bashbrew/flags`, but the base path can be overridden with `--config` or `BASHBREW_CONFIG` (technically, the full path to the `flags` configuration file is `${BASHBREW_CONFIG:-${XDG_CONFIG_HOME:-$HOME/.config}/bashbrew}/flags`). | ||
|
|
||
| ### `bashbrew push` | ||
| To set a default namespace for specific commands only: | ||
|
|
||
| This script takes advantage of `docker login` and does a `docker push` on each `image:tag` specified for the given `--namespaces`. The script will warn if a given `namespace/image:tag` does not exist. | ||
|
|
||
| The `--no-push` option prints out the `docker push` instructions that would have been executed. | ||
| ``` | ||
| Commands: tag, push | ||
| Namespace: officialstaging | ||
| ``` | ||
|
|
||
| ### `bashbrew list` | ||
| To set a default namespace for all commands: | ||
|
|
||
| Takes the same arguments as `bashbrew build` and `bashbrew push`, but prints a list of image names and quits. | ||
| ``` | ||
| Namespace: jsmith | ||
| ``` | ||
|
|
||
| For example: | ||
| A more complex example: | ||
|
|
||
| ```console | ||
| $ # count the number of tags in the official library | ||
| $ bashbrew list --all | wc -l | ||
| 802 | ||
| $ # count the number of _unique_ tags in the official library | ||
| $ bashbrew list --all --uniq | wc -l | ||
| 296 | ||
|
|
||
| $ # pull all officially supported tags of "debian" | ||
| $ bashbrew list debian | xargs -n1 --verbose docker pull | ||
| ... | ||
|
|
||
| $ # list all unique supported tags of "python" | ||
| $ bashbrew list --uniq python | ||
| python:2.7.10 | ||
| python:2.7.10-onbuild | ||
| python:2.7.10-slim | ||
| python:2.7.10-wheezy | ||
| python:3.2.6 | ||
| python:3.2.6-onbuild | ||
| python:3.2.6-slim | ||
| python:3.2.6-wheezy | ||
| python:3.3.6 | ||
| python:3.3.6-onbuild | ||
| python:3.3.6-slim | ||
| python:3.3.6-wheezy | ||
| python:3.4.3 | ||
| python:3.4.3-onbuild | ||
| python:3.4.3-slim | ||
| python:3.4.3-wheezy | ||
| ``` | ||
| # comments are allowed anywhere (and are ignored) | ||
| Library: /mnt/docker/official-images/library | ||
| Cache: /mnt/docker/bashbrew-cache | ||
| Constraints: aufs, docker-1.9, tianon | ||
| ExclusiveConstraints: true | ||
| ## Helper Scripts | ||
| # namespace just "tag" and "push" (not "build") | ||
| Commands: tag, push | ||
| Namespace: tianon | ||
| ### `git-set-mtimes` | ||
| Commands: list | ||
| ApplyConstraints: true | ||
| Commands: tag | ||
| Verbose: true | ||
| ``` | ||
|
|
||
| Since Docker caching of layers is based upon the mtimes of files and folders, this script sets each file's mtime to the time of the commit that most recently modified it and sets each directory's mtime to be the most recent mtime of any file or folder contained within it. This gives a deterministic time for all files and folders in the Git repository. | ||
| In this example, `bashbrew tag` will get both `Namespace` and `Verbose` applied. |
Oops, something went wrong.