Convert "bashbrew" to Go and add a new RFC 2822 manifest file format

To satisfy both Windows (docker-library/golang#92) building and "non-AUFS" (docker-library/official-images#1612, docker-library/official-images#1537) building ("build machine" constraints, if you will), an update to the manifest file format is necessary.

To this end, we're introducing a new format which uses RFC 2822, allowing much more expressivity (especially for additional metadata like `Constraints`) and readability.  The new tool also includes backwards compatibility for the older line-based manifest file format, but its usage is discouraged (and we will be slowly converting at least the `docker-library` repos to use the new format in their `generate-stackbrew-library.sh` scripts).

One of the tangential benefits of this conversion is a massive increase in the speed of the tool (`bashbrew list --all` on a very fast system with an SSD and a flaming hot cache used to take ~5s and now takes ~0.012s).

Author: tianon

README.md

@@ -1,120 +1,81 @@
-# Bashbrew
-
-The recommended way to use `bashbrew.sh` is to install a symlink in your `PATH` somewhere as `bashbrew`, for example `~/bin/bashbrew -> /path/to/official-images/bashbrew/bashbrew.sh` (assuming `~/bin` is in `PATH`).
+# Bashbrew (`bashbrew`)
 
 ```console
 $ bashbrew --help
-
-usage: bashbrew [build|push|list] [options] [repo[:tag] ...]
-   ie: bashbrew build --all
-       bashbrew push debian ubuntu:12.04
-       bashbrew list --namespaces='_' debian:7 hello-world
-
-This script processes the specified Docker images using the corresponding
-repository manifest files.
-
-common options:
-  --all              Build all repositories specified in library
-  --docker="docker"
-                     Use a custom Docker binary
-  --retries="4"
-                     How many times to try again if the build/push fails before
-                     considering it a lost cause (always attempts a minimum of
-                     one time, but maximum of one plus this number)
-  --help, -h, -?     Print this help message
-  --library="/home/tianon/docker/stackbrew/library"
-                     Where to find repository manifest files
-  --logs="/home/tianon/docker/stackbrew/bashbrew/logs"
-                     Where to store the build logs
-  --namespaces="_"
-                     Space separated list of image namespaces to act upon
-                     
-                     Note that "_" is a special case here for the unprefixed
-                     namespace (ie, "debian" vs "library/debian"), and as such
-                     will be implicitly ignored by the "push" subcommand
-                     
-                     Also note that "build" will always tag to the unprefixed
-                     namespace because it is necessary to do so for dependent
-                     images to use FROM correctly (think "onbuild" variants that
-                     are "FROM base-image:some-version")
-  --uniq
-                     Only process the first tag of identical images
-                     This is not recommended for build or push
-                     i.e. process python:2.7, but not python:2
-
-build options:
-  --no-build         Don't build, print what would build
-  --no-clone         Don't pull/clone Git repositories
-  --src="/home/tianon/docker/stackbrew/bashbrew/src"
-                     Where to store cloned Git repositories (GOPATH style)
-
-push options:
-  --no-push          Don't push, print what would push
+NAME:
+   bashbrew - canonical build tool for the official images
+
+USAGE:
+   bashbrew [global options] command [command options] [arguments...]
+
+COMMANDS:
+     list   list repo:tag combinations for a given repo
+     build  build (and tag) repo:tag combinations for a given repo
+     tag    tag repo:tag into a namespace (especially for pushing)
+     push   push namespace/repo:tag (see also "tag")
+
+   plumbing:
+     children  print the repos built FROM a given repo or repo:tag
+     parents   print the repos this repo or repo:tag is FROM
+     cat       print manifest contents for repo or repo:tag
+     from      print FROM for repo:tag
+
+GLOBAL OPTIONS:
+   --verbose, -v            enable more output (esp. "docker build" output) [$BASHBREW_VERBOSE]
+   --no-sort                do not apply any sorting, even via --build-order
+   --constraint value       build constraints (see Constraints in Manifest2822Entry)
+   --exclusive-constraints  skip entries which do not have Constraints
+   --library value          where the bodies are buried (default: "/home/jsmith/docker/official-images/library") [$BASHBREW_LIBRARY]
+   --cache value            where the git wizardry is stashed (default: "/home/jsmith/.cache/bashbrew") [$BASHBREW_CACHE]
+   --help, -h, -?           show help
 
 ```
 
-## Subcommands
-
-### `bashbrew build`
+## Building
 
-This script reads the library files for the images specified and then clones the required Git repositories into the specified `--src` directory. If the Git repository already exists, the script verifies that the Git ref specified in the library file exists and does `git fetch` as necessary.
+Bashbrew itself is built using `gb` ([github.com/constabulary/gb](https://github.com/constabulary/gb)).
 
-The next step in the script is to build each image specified. All the `image:tag` combinations are placed into a queue. The processing order is determined by the order of items passed in on the command line (or alphabetical if `--all` is used). When a whole image, like `debian`, is specified the `image:tag` combinations are added to the queue in the order that they appear in the library file. For each `image:tag` to be processed, the system checks out the specified commit and sets mtimes (see [`git-set-mtimes`](#git-set-mtimes)) of all files in the Git repository to take advantage of Docker caching. If the `image:tag` is `FROM` another image that is later in the queue, it is deferred to the end of the queue.
+Once in the `go` subdirectory, `gb build` should produce `go/bin/bashbrew`, ready for use.
 
-After the image is built, the final step of the script is to tag the image into each of the given `--namespaces`.
+## Usage
 
-The `--no-clone` option skips the `git clone` step and will cause the script to fail on the build step if the Git repository does not exist or is missing the required Git refs.
+In general, `bashbrew build some-repo` or `bashbrew build ./some-file` should be sufficient for using the tool at a surface level, especially for testing. For more complex usage, please see the built-in help (`bashbrew --help`, `bashbrew build --help`, etc).
 
-The `--no-build` option skips all the building, including setting the mtimes.
+## Configuration
 
-**WARNING:** setting `--src` so that it uses a local working copy of your Git directory for a specified build will delete untracked and uncommitted changes, and will disable `gc.auto`. It is not recommended to symlink in your working directories for use during build.
+The default "flags" configuration is in `~/.config/bashbrew/flags`, but the base path can be overridden with `--config` or `BASHBREW_CONFIG` (technically, the full path to the `flags` configuration file is `${BASHBREW_CONFIG:-${XDG_CONFIG_HOME:-$HOME/.config}/bashbrew}/flags`).
 
-### `bashbrew push`
+To set a default namespace for specific commands only:
 
-This script takes advantage of `docker login` and does a `docker push` on each `image:tag` specified for the given `--namespaces`. The script will warn if a given `namespace/image:tag` does not exist.
-
-The `--no-push` option prints out the `docker push` instructions that would have been executed.
+```
+Commands: tag, push
+Namespace: officialstaging
+```
 
-### `bashbrew list`
+To set a default namespace for all commands:
 
-Takes the same arguments as `bashbrew build` and `bashbrew push`, but prints a list of image names and quits.
+```
+Namespace: jsmith
+```
 
-For example:
+A more complex example:
 
-```console
-$ # count the number of tags in the official library
-$ bashbrew list --all | wc -l
-802
-$ # count the number of _unique_ tags in the official library
-$ bashbrew list --all --uniq | wc -l
-296
-
-$ # pull all officially supported tags of "debian"
-$ bashbrew list debian | xargs -n1 --verbose docker pull
-...
-
-$ # list all unique supported tags of "python"
-$ bashbrew list --uniq python
-python:2.7.10
-python:2.7.10-onbuild
-python:2.7.10-slim
-python:2.7.10-wheezy
-python:3.2.6
-python:3.2.6-onbuild
-python:3.2.6-slim
-python:3.2.6-wheezy
-python:3.3.6
-python:3.3.6-onbuild
-python:3.3.6-slim
-python:3.3.6-wheezy
-python:3.4.3
-python:3.4.3-onbuild
-python:3.4.3-slim
-python:3.4.3-wheezy
 ```
+# comments are allowed anywhere (and are ignored)
+Library: /mnt/docker/official-images/library
+Cache: /mnt/docker/bashbrew-cache
+Constraints: aufs, docker-1.9, tianon
+ExclusiveConstraints: true
 
-## Helper Scripts
+# namespace just "tag" and "push" (not "build")
+Commands: tag, push
+Namespace: tianon
 
-### `git-set-mtimes`
+Commands: list
+ApplyConstraints: true
+
+Commands: tag
+Verbose: true
+```
 
-Since Docker caching of layers is based upon the mtimes of files and folders, this script sets each file's mtime to the time of the commit that most recently modified it and sets each directory's mtime to be the most recent mtime of any file or folder contained within it. This gives a deterministic time for all files and folders in the Git repository.
+In this example, `bashbrew tag` will get both `Namespace` and `Verbose` applied.