README.build

This file documents how to build the Crowbar installation DVD image.

Prerequisites:
  * An unfiltered Internet connection and a decent amount of bandwidth.
    The first time you try to build Crowbar it will need to download
    all the packages that it will need to stage Crowbar onto an OS install
    DVD.
  * bash version 4 or higher
    build_crowbar.sh uses associative arrays, which bash got in version 4.
  * mkisofs
    The end product of the build script is an ISO image that will be used
    to bootstrap the Crowbar admin node. 
  * debootstrap (if staging on to Ubuntu)
    The build process needs to download all the .debs and gems that 
    Crowbar requires, and we don't want to inadvertently mess up the build
    machine when we do that.  All extra packages are downloaded into a
    chrooted minimal Ubuntu intall, and we use debootstrap to enable that.
  * Sudo to root privileges for the following commands:
    * /bin/mount, /bin/umount
      We have to be able to mount and umount the Ubuntu .iso image, as well as
      a tmpfs for debootstrap, and we have to be able to bind mount 
      /dev, /dev/pts, /proc, and /sys into the debootstrap chroot environment.
    * /usr/sbin/debootstrap
      debootstrap requires root privileges to run.
    * /bin/cp
      We need to copy things into and out of the debootstrap environment to
      ensure it downloads and caches the right packages.
    * /usr/sbin/chroot
      All our package caching is done in a chroot environment, and chroot
      requires root permissions to run.

      If you want to allow build_crowbar to run the above commands as root
      without having to enter a password each time, the build_crowbar.sh
      includes a sample line you can fix up and add to /etc/sudoers.

Command Line Parameters:
  * The first command line parameter is the OS you want to 
    stage Crowbar on to.  Crowbar currently understands how to stage on to 
    ubuntu-10.10, redhat-5.6, redhat-5.7, and centos-5.7

  --update-cache
    This parameter forces the build to try and update the build cache again.  
    Use this parameter if you want to pull in updates from the upstream 
    repositories that Crowbar pulls packages from.

  --merge|-m
    Any arguments after this that do not start with a hyphen are interpreted as
    branches in the git repository that you want to merge along with the current
    branch into a throwaway branch before staging Crowbar on to the build .iso.
    If you are working on developing a barclamp or have changes in other 
    branches that you want to test out, this is the option for you. If your 
    build cache is also a git repository, it will try to merge identically
    named branches in that repository as well -- this is a good way of testing
    new packages before signing off on them. After the build is finished, the
    build script will delete any throwaway branches it created, leaving the
    repositories in the same state they started in.
  
  --barclamps
    Any arguments after this that do not start with a hyphen are interpreted
    as extra barclamps to stage on the final .ISO.  The build system will use
    the metadata in the crowbar.yml files to figure out all the dependent
    barclamps of the ones on the command line and include those as well.  You
    can also lass a barclamp group name my prefixing it with an @ sign, and
    the build system will expand that into its component barclamps.

  --test
    This tells the build system to try and smoketest the freshly-generated
    .ISO. Any arguments after this that do not begin with a hyphen are
    passed to the test framework -- please see test_framework/README.testing
    for more information.

  --ci <barclamp> <branch>
    This tells the build system to test <branch> of <barclamp> to see if it
    merges cleanly in with the current codebase and passed the smoketest. 
    If it does, the build system will finalize the merge and update the
    submodule reference in the main crowbar repository, otherwise it will 
    roll back the merges and fail.  This option is intended to be used by
    Jenkins to implement gated submodules for the Crowbar repository.

  --shrink
    This asks the build system to throw out unneeded packages from the base OS
    install media.  It relies on the presence of a minimal-install file listing
    the minimal set of packages needed to deploy an admin node.  Currently,
    we only know how to do this on rpm based systems, because Ubuntu installs
    are fairly small to begin with anyways.

  --no-cache-update
    This tells the build system that it should die instead of trying to update
    the build cache.  This is useful for automated build processes where you want
    to build Crowbar with a curated set of packages.  When --no-cache-update is
    present and the build cache is a Git repository, the current checkout of the
    build cache will be recorded in the build-info file.

  --no-iso
    Do everything but actaully build the iso.  This is useful for updating
    any caches that need updating without actaully forcing a cache update.

  --wild-cache
    Create a temporary cache directory that is good for this build
    only. The cache will be deleted when the build exits.  This exists
    mainly to test what a build that is pulling the latest stuff from
    the Internet looks like.  You should make sure that you are using
    a caching web proxy such as polipo or squid, otherwise repeated
    builds will be very slow.

  --do-not-clean
    Skip using git clean to clean up the trees before starting the build.
    Only use this flag if you can tolerate build breaks due to having
    files present that should not be there, and you are confortable with
    debugging that state yourself.  If you are not comfortable using
    this flag, you should be using dev build.

Usage:
  * Run build_crowbar.sh from the Crowbar git repository.  
    The first time it is run, it will download and cache all the files it 
    needs to stage Crowbar on to the OS installation DVD, build an OS install
    ISO with Crowbar staged on to it, save the generated .ISO to the current 
    working directory, and print out a message saying where it saved the image.
    On subsequent runs it will run with the files it cached from the first
    run, unless update-cache is passed as a parameter to the script.

Customization:
  
   build_crowbar.sh has several different parameters you can tune, either from
   $HOME/.build-crowbar.conf (for developer use), or from build-crowbar.conf
   in the current directory (for automated builds).

   Here are the parameters you can change through the above configuration files:
   * DEBUG
     If DEBUG is set to anything, build_crowbar will run in debug mode, and will
     print a transcript of everything it is doing to standard error.
   * CACHE_DIR
     This is the default location where build_crowbar.sh will keep the files
     it caches, along with the temporary directories used to mount the
     ISO image, the debootstrap chroot, and the directory we perform the build
     in.  It defaults to $HOME/.crowbar-build-cache.
   * ISO_LIBRARY
     This is the default location where the Ubuntu .iso is stored. It defaults 
     to $CACHE_DIR/iso
   * ISO_DEST
     This is the location that we will save the Crowbar install image to.
     It defaults to the current directory.
   * IMAGE_DIR
     This is the location that we will mount isos in.  
     It defaults to $CACHE_DIR/image
   * SLEDGEHAMMER_PXE_DIR
     This points to the location we expect to find the unpacked Sledgehammer
     PXE boot archive.  It defaults to $CACHE_DIR/tftpboot
   * VERSION
     The default version of Crowbar.  Defaults to dev.
   * BUILT_ISO
     The name of the ISO that build_crowbar.sh generates.
     Defaults to crowbar-$VERSION.iso
   * CROWBAR_DIR
     The directory that the Crowbar source is cheched out to.
     Defaults to the directory that build_crowbar.sh is in.
   * VCS_CLEAN_CMD
     This is the command that build_crowbar.sh will run to clean the tree before
     staging the Crowbar build.  
     Defaults to 'git clean -f -d'

Build System Walkthrough:

When build_crowbar.sh is invoked, it performs the following processes in order:

 1: Make sure we are in the C locale, and that $PATH is set to something sane.
 2: Pick up any local configuration settings from $HOME/.build_crowbar.conf
    or ./build_crowbar.conf
 3: If USE_PROXY is set to 1, make sure that the HTTP and HTTPS proxy
    settings are sane.
 4: Set any uninitialized config variables to their defaults.
 5: Source our generic build and test functions.
 6: Figure out what OS we want to stage, and source the build and test
    libraries.  This will pull in the functions we need to actually stage
    Crowbar on an OS install ISO.  If we were asked to build on an OS that 
    we don't have build info for, die and print out the OSes we do know how
    to stage things on.
 7: Make sure that all the commands we will need to stage Crowbar on to an ISO
    are installed on the system.  If they are not, print a helpful error 
    message and die.
 8: Grab the build lock to make sure that multiple builds do not stomp all
    over eachother.
 9: Do a little bookkeeping to make sure we are on a buildable Git branch.
    If the build cache is in a git repository, record that information as well.
10: Parse our commandline options. 
11: Make sure our essential build-related directories are present (including
    the directory we will stage the build into, the directory we will mount
    the ISO image on, and a chroot that will be used as part of the barclamp
    staging process), and set up any build parameters that have not already 
    been set up.
12: If we were not passed a list of barclamps to install on the command line,
    figure out what barclamps we need based on the submodule information from
    the git branch we are on.
13: Pull in metadata from the crowbar.yml files for each barclamp.
    This metadata will drive the rest of the install -- we need it to
    figure out dependency relations between barclamps, what packages and files
    to stage, and how to invoke any external build processes we might need.
14: Make sure we have a Sledgehammer image handy, and build it if we don't.
15: If we don't have the OS ISO to stage on, and we know how to get one, then
    download the .ISO we will need.
16: Clean out any leftovers from the last build, and make sure that
    we don't inadvertently pull in any VCS cruft.
17: Mount the OS iso as a loopback file system, and index its package
    pool if we don't already have it cached.
18: Stage some barclamp-independent build information into the build directory.
19: Create the build-info file for this build, and start adding useful 
    metadata into it.
20: Loop over the list of barclamps want to stage, and stage each one.  
    This is covered in more detail in the Barclamp Staging Walkthrough below.
21: Bundle each barclamp and its package cache into a per-barclamp tarball.
22: Create some legacy symlinks, and stage any custom proposals that this
    iso will use.
23: Perform any OS specific fixups that are needed to make this image deploy
    correctly.
24: Stage the Sledgehammer image.
25: Create the Crowbar .ISO by merging the contents of the build and the image
    directory.  Wherever there is a conflict in file names or contents, the
    build directory has priority.  If we were asked to generate a shrunken ISO,
    that happens here.
26: If we were asked to test the ISO, invoke the test framework on our 
    newly-created ISO.
27: Clean up after ourselves.

Barclamp Staging Walkthrough:

build_crowbar will try to stage each barclamp in dependency order (as inferred
from each barclamp's crowbar.yml file).  Staging a barclamp properly for the
OS we are staging on to requires the use of a chroot environment to ensure
that we get all the packages we need and that we don't break the host OS
in the process.  Each barclamp is staged in 6 phases:

1: Check to see if all the OS packages listed in the pkgs: and build_pkgs: 
   section of the crowbar.yml are present in this barclamps's OS build cache.
   If they are not, fetch any missing ones and all their dependencies using
   a chroot environment, and add any new or updated packages so fetched back
   into the build cache.
2: Check to see that all the gems listed in the crowbar.yml are present in
   the build cache.  If they are not, fetch them and all their dependencies in
   the chroot environment, and add any new or updated gems to the build cache.
3: Download and cache any packages pointed to by raw_pkg stanzas in the 
   crowbar.yml that we are missing.
4: Download and cache any files required by extra_files stanzas in the
   crowbar.yml thatwe are missing.
5: If the crowbar.yml has a build_cmd stanza, source that file and use it
   to build an external package.  The script pointed to by build_cmd should
   have two functions declared:
  
   bc_needs_build -- This function should return 0 if the external
   pacakge needs building, and 1 if it does not.
   
   bc_build: This function will be invoked after setting up a chroot and
   bind-mounting the build cache for this barclamp into it.  It is responsible
   for using the chroot enviromnent to build the external package, and making
   sure that the output of the build process winds up in the proper location
   in the build cache so that the rest of the barclamp can properly use it.

   bc-build has access to the following environment variables:
   BC_DIR = the full path to the root of the barclamp source repository.  
   BC_CACHE = The full path to the barclamp package cache.

   Any actual building should happen in the chroot environment. 
   To facilitate this, $BC_CACHE is bind-mounted to /mnt in the chroot,
   any build_pkgs required by this barclamp will be installed in the chroot,
   and /mnt/current_os in the chroot will be a symlink to the OS package cache
   that for the barclamp build cache that is bind mounted to /mnt in the chroot.

   You can use the chroot_install command to install any additional packages
   you may need, and you can use the in_chroot command to run commands
   in hte chroot environment.

   For some in-tree examples, refer to the ganglia, provisioner, and deployer
   barclamps.  All of these have an additional script that is copied into the
   chroot that handles most of the build tasks.

6: Tar up the source for the barclamp and the build cache into a deployable
   barclamp tarball.
   

The build_crowbar.sh script is heavily commented, please refer to it for more 
detailed information.