requirements | batteries included | toys in the toybox

optional stuff | tests | demos | implementation remarks | contributing | quick-links | todo | credits

##Toybox

Toybox is a template that builds an awesome development environment for virtualbox using vagrant and puppet. Apart from actually using the toybox as a development playground, this should be usable as pattern boilerplate for other kinds of custom automation. Toybox includes demos and integration tests for the infrastructure that is bootstrapped, and the tests can be run from either the guest or host OS. ##Build target/reference versions:

Vagrant 1.6.5 and Virtualbox 4.3.18 on the host with an Ubuntu guest. Note that currently only some random subset of the toys will work with redhat/centos. At least across recent Ubuntu versions, the puppet code is probably (possibly? maybe? hopefully??) fairly generic.

A known working base-box is Ubuntu 14.04 "trusty" (for download command, see "Usage" section). Download vagrant here, download virtualbox here.

##Basic batteries are included: * misc: git, ack-grep, nmap, screen, and tree * basic dev: * ruby: ruby 1.9.3, ruby-dev, gem * python: python 2.7.6, python-pip, python-dev, python-virtualenv ##Other toys in the toybox * [mongodb](http://www.mongodb.org): a popular nosql database * default port @ 27017 * mongo version is 2.4.9 * started by default on system boot * [genghisapp](http://genghisapp.com): a data viz tool for mongo * version @ 2.3.11 * WUI port at 5556 * see Vagrantfile to check if port-forwarding is enabled * [rabbitmq](https://www.rabbitmq.com): a message queue * Erlang R16B03 will be installed * data port @ 5672 * rabbit version is 3.4.0 * users: admin/admin, guest/guest * WUI port at 15672 * see Vagrantfile to check if port-forwarding is enabled * [celery](http://celery.readthedocs.org): task queue framework (uses rabbit) * celery version is 3.1.6 * no workers are defined or started by default * [flower](http://flower.readthedocs.org/en/latest/): a data viz tool for celery * flower version is 0.7.3 * WUI port @ 5555 * see Vagrantfile to check if port-forwarding is enabled * [supervisor](http://supervisord.org): a configurable, lightweight daemon tool * version @ 3.0 * responsible for daemonizing flower * responsible for daemonizing genghisapp * WUI port @ 9001 * see Vagrantfile to check if port-forwarding is enabled * [nginx](http://nginx.org/en/docs/): a webserver with fairly sane configs * version @ 1.4.6 * WUI/data port @ 80 * default config is simple: just a rendered version of this markdown * see Vagrantfile to check if port-forwarding to host 8080 is enabled

##Optional toys:

• neo4j: a graph database
  • WUI/data port @ 7474
  • gDB kernel version is 1.7.2
  • this will also install Java. sorry
  • see Vagrantfile to check if port-forwarding is enabled
• XWindows and XMonad 0.11
  • basic naked installation (well ok, and emacs23)
  • you must use startx; will not startup on boot
  • the sole window manager (XMonad) will be used by default with "startx".
  • xmonad implicitly requires haskell

## Quick Start

Clone the repository and cd into it

  $ git clone https://github.com/mattvonrocketstein/toybox.git
  $ cd toybox

Bootstrap vagrant with the plugins and basebox toybox uses

  $ vagrant box add trusty64 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-i386-vagrant-disk1.box
  $ vagrant plugin install vagrant-vbguest
  $ vagrant plugin install vagrant-scp

Bring up the new toybox and provision it. This will take a long time while the basebox is bootstrapped and the provisioning downloads package updates. Also Note that if vbguest detects a mismatched guest-additions iso, it may tke even longer while it corrects this. Subsequent calls like the one below will not by default rerun provisioning.

  $ vagrant up

If the command above happens to terminate with a red message "The SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed", it might be caused by some download via apt, gem, or pip which has timed out. If that's the case, you may want to retry provisioning by executing vagrant provision.

If you don't see the "non-zero exit status" message, then it probably succeeded. But just to be sure that puppet hasn't missed starting or restarting a service that it's updated, you should run

  $ vagrant reload

If you got this far, your new box should alreaady be setup and working. You can connect to it now with the vagrant ssh command or try running the tests.

You might want to send over some ssh keys to your fresh new development box. To copy everything except for the local autothorized_keys into your toybox,

  $ find ~/.ssh -type f|grep -v authorized_keys|xargs -I{} vagrant scp {} /home/vagrant/.ssh/

##Advanced Usage: Optional Provisioning The optional items are optional mostly because they are big. You probably don't want this stuff to slow down your install on a slow connections or headless box.

####Provisioning XWindows

If you wish to use the optional xwindows setup, I strongly suggest installing the vbguest plugin vbguest plugin. (Note: Unfortunately even then getting full-screen resolution to work may still take some extra fiddling, the situation seems to change slightly with every minor-version release of guest-extensions/virtualbox.) If you want to change the window manager or other details of this aspect of provisioning, fork this repo and edit toybox/puppet/modules/site/manifests/xwindows.pp. To enable provisioning for xwindows, run:

  $ PROVISION_XWIN=true vagrant provision

####Provisioning Neo

Provisioning neo is similar to provisioning XWindows, but you will need to download their distribution tarball first. Note: before you start the download, make sure that you're still in the same directory as this README and the Vagrantfile.

  $ wget http://dist.neo4j.org/neo4j-community-1.7.2-unix.tar.gz
  $ PROVISION_NEO=true vagrant provision

##Running Tests Tests can be run from either the guest or the host, but the meaning of each is slightly different. Tests will autodetect whether they are running from the guest or the host based on the presence of the `/vagrant` directory.

By default, the Vagrantfile forwards lots of ports for the services puppet is expected to bring up. During development it can be useful to verify that those services are indeed alive. To bootstrap the testing-setup on the host:

  $ virtualenv host_venv
  $ source host_venv/bin/activate
  $ pip install -r tests/requirements.txt
  $ python tests/test_guest.py

During normal provisioning, guest_venv is setup automatically. To run tests on the guest from the guest, run this command from the host:

  $ vagrant ssh -c "/vagrant/guest_venv/bin/python /vagrant/tests/test_guest.py"

##Running Demos During default provisioning, databases, message queues, and visualization aids are setup but there is no data to populate them. Demos included with toybox are just code examples to create some traffic. All demos require you to connect to the guest and source the main guest virtual-environment:

  $ vagrant ssh # connect to guest
  $ source /vagrant/guest_venv/bin/activate # run this from guest

To run the celery/rabbit demo follows the instructions below. You can confirm the operations by watching graphs change in real time on your local flower and rabbitmq servers.

  # send 1000 and 500 tasks to add and subtract worker, respectively
  $ python /vagrant/demos/demo_celery.py --add -n 1000
  $ python /vagrant/demos/demo_celery.py --add -n 500
  # start a worker to deal with tasks
  $ python /vagrant/demos/demo_celery.py --worker

To run the MongoDB demo follow the instructions below. You can confirm the operations by checking your local genghisapp, specifically the user collection.

  # create 50 fake users
  $ python /vagrant/demos/demo_mongo.py --records 50

To run the Neo4j demo you must already have done some of the optional provisioning, and then you can follow the instructions below. If it's not present on the guest in the /vagrant directory, the example movies database will be downloaded and afterwards it will be loaded into your neo server. After loading a dataset, visit your local neo server. If you want to start over, you can flush the database by using the --wipedb argument to the demo_neo.py script. See the script code for other usage instructions.

  # load default datset "cieasts_12k_movies_50k"
  $ python /vagrant/demos/demo_neo.py

##Implementation Remarks This section documents a few things that might be useful to people forking this recipe. If you need toybox to execute additional git-clones, create or provision python virtualenvironments, etc, the examples in **[1]** will be useful. To modify the nginx setup, start in **[2]**. To execute additional configuration in the very last step of provisioning, see **[3]**. To change the window-manager or other applications installed in provisioning xwindows, see **[4]**. For examples of daemonizing random processes, check out the supervisorctl section in **[5]**. As a place to add default additional system packages, **[6]** is the suggested spot.

1. puppet/modules/site/manifests/my_code.pp
2. puppet/modules/site/files/nginx_conf/sites-enabled/default
3. puppet/modules/site/manifests/configuration.pp
4. puppet/modules/site/manifests/xwindows.pp
5. puppet/modules/core/manifests/toybox.pp
6. puppet/modules/core/manifests/basic_dev.pp

####Pattern Idempotency Much effort has gone into making toybox as friendly as possible for low-bandwidth situations. During repeated calls to `vagrant provision`, every effort has been made to avoid unnecessary duplication of effort for expensive network operations like `apt-get update`, `git clone`, and `pip install`. However, relevant changes to configuration that involve new packages or changes to template files, etc, should always be honored. Please file an issue on github if you find problems. ####Puppet File Layout * Entry-point is `puppet/default.pp` (as named in the Vagrantfile) * This is probably your starting place for fork-and-mod hacks * The `puppet` directory has two subdirs, namely `core` and `site` * `core` is meant for modules that will not change. * `site` is meant for modules that are more idiosyncratic * Two stages are defined in `default.pp`: 'first' and 'last'. * Stages are used to guarantee aspects of the run-order * see [puppet language guide](http://docs.puppetlabs.com/guides/language_guide.html) and the section "Run Stages" ##Contributing:

Issues can be raised on the bugtracker and pull requests are always welcome.

##Quick links: This markdown file is rendered to html and used as the default landing page for the toybox nginx installation. If you're looking at that page, you might find the following links useful:

• genghis
• flower
• rabbitmq
• supervisor
• nginx
• neo

##TODO: * Experimentation with the [AWS provider](https://github.com/mitchellh/vagrant-aws)? * optional install for gephi (a graphdb browser)? use [these instructions](https://gist.github.com/dcht00/432caaf3e6c50a2202b8) * elasticsearch? * zookeeper? ##Credits: Puppet can sometimes make it pretty difficult it is to reuse other code without forking. Apart from puppet forge standard libraries included in this repo and amongst other things, I have benefited from the work mentioned below:

• https://github.com/opencredo/neo4j-puppet
• https://github.com/aubricus/vagrant-puppet-boilerplate
• https://forge.puppetlabs.com/proletaryo/supervisor
• https://github.com/netmanagers/puppet-nginx
• https://github.com/nesi/puppet-git