Merge branch 'master' of github.com:rapid7/lab

Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in lab.gemspec
+gemspec

Gemfile.lock

@@ -0,0 +1,21 @@
+PATH
+  remote: .
+  specs:
+    lab (0.1.0)
+      net-scp
+      net-ssh
+      nokogiri
+
+GEM
+  remote: http://rubygems.org/
+  specs:
+    net-scp (1.0.4)
+      net-ssh (>= 1.99.1)
+    net-ssh (2.3.0)
+    nokogiri (1.5.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  lab!

README.md

@@ -0,0 +1,80 @@
+This folder contains the libraries necessary to run the lab plugin, and can also be used in a standalone way to automate virtual machines.
+
+CONCEPTS:
+=========
+
+The lab provides a clean interface to common vm functions such as start / stop / snapshot / revert and even running system commands or higher-level functions like opening a browser to a specified URL. It's designed so the different VM technologies have a similiar interface, and you can ignore the specifics of the VM tech. The majority of the functionality is implemented in the form of drivers and controllers. Drivers implement the underlying command for each vm software (such as start/stop/revert), and controllers implement the commands which apply to all vms (such as listing all running vms, or cloning a vm). 
+
+If you're interested in porting a vm software (see below), please take a look at the workstation_driver.rb and the workstation_controller.rb -- This is a simple driver / controller in the lab, and you can simply copy / modify this to implement a new driver & controller for the software. 
+
+SUPPORTED VM TECHNOLOGIES:
+==========================
+NOTE: The lab libraries have only been tested with linux as a host, porting to windows is not planned at this time.
+
+Implemented:
+ - workstation (Tested against 7.x)
+ - remote_workstation (Tested against 7.x)
+ - virtualbox (Tested against 4.x)
+ - remote_esx (VMware ESX Host Agent 4.1.0 build-348481)
+
+Partially Implemented:
+ - amazon_ec2 (via fog gem)
+ - dynagen
+
+Need Implementation:
+ - qemu
+ - qemudo
+ - others?
+
+PLATFORM SUPPORT:
+=================
+You will need to have this code running on a linux box, Currently this has only been run / tested on Ubuntu 9.04 -> 10.04, though it should run on any linux with an ssh client and the dependencies below. Remote VM Hosts will need to be linux as well, though other platforms may work (untested). If you're interested in porting it to windows, please contact me (jcran). 
+
+Platform Dependencies:
+  - whatever vm software is necessary for the driver you're using (see SUPPORTED VM TECHNOLOGIES above)
+  - net/scp - the gem (net-scp). Required to copy files to/from the devices in the case that tools are not installed. Not necessary if tools are installed. 
+  - fog - require to use the amazon_ec2 driver
+
+STANDALONE API:
+===============
+BACKGROUND:
+
+The lab libraries add tons of useful functionality that isn't exposed through the lab plugin, such as the ability to run commands on hosts. This library can serve as an excellent base for more complex operations on a remote host as well. 
+
+USAGE:
+
+You must first create a yaml file which describes your vm. See data/lab/test_targets.yml for an example.  
+<pre>
+  require 'vm_controller'
+  vm_controller = ::Lab::Controllers::VmController.new(YAML.load_file(lab_def)) 
+  vm_controller['vm1'].start
+  vm_controller['vm1'].snapshot("clean") 
+  vm_controller['vm1'].run_command("rm /etc/resolv.conf")
+  vm_controller['vm1'].open_uri("http://autopwn:8080")
+  vm_controller['vm1'].revert("clean")
+  vm_controller['vm1'].revert("clean")
+</pre>
+METASPLOIT MSFCONSOLE LAB PLUGIN:
+=================================
+
+BACKGROUND:
+
+The lab plugin for msfconsole adds a number of commands which may be useful if you're interested in automating remote hosts with rc scripts, or if you need to control targets / support systems while utilizing the metasploit console. A potential use case is testing an IPS / IDS, and resetting the target after running each exploit. 
+
+USAGE:
+
+Here's some example usage for the lab plugin. 
+<pre>
+  msf> load lab                           // Loads the lab plugin
+  msf> lab_load <path_to_lab_file>        // Loads from a lab configuration file. See data/lab/test_targets.yml for an example
+  msf> lab_load_dir workstation /path/to/vmx/files  // Loads from a local directory.
+  msf> lab_load_running remote_esx root esx_server  // Loads all running vms. 
+  msf> lab_start vm1                      // Start a vm which was loaded above
+  msf> lab_snapshot vm1 snapshot_1        // Snapshot a vm as 'snapshot_1'
+  msf> lab_run_command ("rm -rf /")       // oops!
+  msf> lab_show                           // Show all vms that we're aware of
+  msf> lab_show_running                   // Show only running vms
+  msf> lab_start vm2                      // Start another vm
+  msf> lab_suspend  vm1                   // Suspend a vm
+  msf> lab_revert all snapshot_1          // Revert all vms back to 'snapshot_1'
+</pre>

Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

TODO

@@ -0,0 +1,15 @@
+This is a list of basic priorities for the lab code...
+
+* Implement more technologies
+
+ a) finish amazon ec2 (via fog)
+ b) qemu
+ c) qemudo
+ d) kvm
+ e) other cloud technologies (newservers, slicehost/rackspace,etc)
+
+* Implement a cloning function on each controller
+
+* Support Windows as a host platform. Currently all the code assumes a linux host is running it. The same applies for the remote_* drivers -- they've not been tested on windows.
+
+* Consolidate the remote_system_command code & provide a filter. Create an unsafe_system_command and unsafe_remote_system_command function call for when we control the entire string.

config/test_lab.yml

@@ -0,0 +1,11 @@
+ - vmid: backtrack
+   driver: workstation
+   location: /opt/vm/backtrack5/Backtrack5x64.vmx
+   modifiers:
+     - Test
+   credentials:
+     - user: root
+       pass: toor
+   os: linux
+   flavor: ubuntu
+   arch: 64

config/test_targets.yml

@@ -0,0 +1,21 @@
+ - vmid: metasploitable
+   driver: workstation
+   location: /opt/vm/lab/user/Metasploitable/Metasploitable.vmx
+   tools: false
+   credentials: 
+     - user: msfadmin
+       pass: msfadmin
+ - vmid: windows2000_target
+   driver: workstation
+   location: /opt/vm/lab/vuln/msf_Win2000SP4/Windows 2000 AS.vmx
+   tools: true
+   credentials: 
+ - vmid: windowsxp_target
+   driver: remote_workstation
+   host: vmhost
+   user: root
+   location: /opt/vm/lab/vuln/msf_WinXPSP1/Windows XP Professional.vmx
+   tools: true
+   credentials: 
+     - user: administrator
+       pass: administrator

lab.gemspec

@@ -0,0 +1,35 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib/', __FILE__)
+$:.unshift lib unless $:.include?(lib)
+require 'lab/version'
+
+Gem::Specification.new do |s|
+  s.name        = "lab"
+  s.version     = Lab::VERSION
+  s.authors     = ["Jonathan Cran"]
+  s.email       = ["jcran@rapid7.com"]
+  s.homepage    = "http://www.github.com/rapid7/lab/wiki"
+  s.summary     = %q{Manage VMs like a boss}
+  s.description = %q{Start/Stop/Revert and do other cool stuff w/ Vmware, Virtualbox, and ESXi vms. This gem wraps common CLI utilities and other gems to create a common inteface for vms. }
+
+  s.rubyforge_project = "lab"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  # specify any dependencies here; for example:
+  # s.add_development_dependency "rspec"
+
+    # ??
+    s.add_runtime_dependency "nokogiri"
+
+    # Multiple things - fallback execute / copy
+    s.add_runtime_dependency "net-ssh"
+    s.add_runtime_dependency "net-scp"
+
+    # Vmware ESX driver
+    s.add_runtime_dependency "rbvmomi"
+
+end

lib/lab.rb

@@ -0,0 +1,2 @@
+require 'lab/vm_controller'
+require 'lab/version'

lib/lab/controller/dynagen_controller.rb

@@ -0,0 +1,14 @@
+module Lab
+module Controllers
+module DynagenController
+
+  def self.running_list
+    raise "Unsupported"
+  end
+
+  def self.dir_list(basepath=nil)
+    raise "Unsupported"
+  end
+end
+end
+end

lib/lab/controller/fog_controller.rb

@@ -0,0 +1,6 @@
+module Lab
+module Controllers
+module FogController
+end
+end
+end

lib/lab/controller/remote_esx_controller.rb

@@ -0,0 +1,62 @@
+# This controller was built against: 
+# VMware ESX Host Agent 4.1.0 build-348481
+
+module Lab
+module Controllers
+module RemoteEsxiController
+
+  # Note that 3.5 was different (vmware-vim-cmd)
+  VIM_CMD = 'vim-cmd'.freeze
+
+  def self.dir_list(basepath=nil)
+    # Does this method really even make sense for esx?
+    return "Unsupported :("
+  end
+
+  def self.running_list(user, host)
+    user.gsub!(/(\W)*/, '')
+    host.gsub!(/(\W)*/, '')
+
+    # first get all registered vms
+    registered_vms = self.get_vms(user, host) || []
+    running_vms = []
+
+    # now let's see which ones are running
+    # TODO:  this is ghetto, would be better not to connect repeatedly
+    registered_vms.each do |vm|
+      remote_cmd = "ssh #{user}@#{host} \"#{VIM_CMD} vmsvc/power.getstate #{vm[:id]}\""
+      raw = `#{remote_cmd}`
+      running_vms << vm if raw =~ /Powered on/
+    end
+
+    return running_vms
+  end
+
+private 
+
+  def self.get_vms(user, host)
+    user.gsub!(/(\W)*/, '')
+    host.gsub!(/(\W)*/, '')
+
+    vms = [] # array of VM hashes
+    remote_cmd = "ssh #{user}@#{host} \"#{VIM_CMD} vmsvc/getallvms | grep ^[0-9] | sed 's/[[:blank:]]\\{3,\\}/ /g'\""
+    raw = `#{remote_cmd}`.split("\n")
+
+    raw.each do |line|
+      # So effing ghetto
+      id_and_name = line.split('[datastore').first
+      id = id_and_name.split(' ').first
+
+      ## TODO - there's surely a better way to do this.
+      name_array = id_and_name.split(' ')
+      name_array.shift
+      name = name_array.join(' ')
+      vms << {:id => id, :name => name}
+    end
+
+    return vms
+  end
+
+end
+end
+end

lib/lab/controller/remote_esxi_controller.rb

@@ -0,0 +1,62 @@
+# This controller was built against: 
+# VMware ESX Host Agent 4.1.0 build-348481
+
+module Lab
+module Controllers
+module RemoteEsxiController
+
+  # Note that 3.5 was different (vmware-vim-cmd)
+  VIM_CMD = 'vim-cmd'.freeze
+
+  def self.dir_list(basepath=nil)
+    # Does this method really even make sense for esx?
+    return "Unsupported :("
+  end
+
+  def self.running_list(user, host)
+    user.gsub!(/(\W)*/, '')
+    host.gsub!(/(\W)*/, '')
+
+    # first get all registered vms
+    registered_vms = self.get_vms(user, host) || []
+    running_vms = []
+
+    # now let's see which ones are running
+    # TODO:  this is ghetto, would be better not to connect repeatedly
+    registered_vms.each do |vm|
+      remote_cmd = "ssh #{user}@#{host} \"#{VIM_CMD} vmsvc/power.getstate #{vm[:id]}\""
+      raw = `#{remote_cmd}`
+      running_vms << vm if raw =~ /Powered on/
+    end
+
+    return running_vms
+  end
+
+private 
+
+  def self.get_vms(user, host)
+    user.gsub!(/(\W)*/, '')
+    host.gsub!(/(\W)*/, '')
+
+    vms = [] # array of VM hashes
+    remote_cmd = "ssh #{user}@#{host} \"#{VIM_CMD} vmsvc/getallvms | grep ^[0-9] | sed 's/[[:blank:]]\\{3,\\}/ /g'\""
+    raw = `#{remote_cmd}`.split("\n")
+
+    raw.each do |line|
+      # So effing ghetto
+      id_and_name = line.split('[datastore').first
+      id = id_and_name.split(' ').first
+
+      ## TODO - there's surely a better way to do this.
+      name_array = id_and_name.split(' ')
+      name_array.shift
+      name = name_array.join(' ')
+      vms << {:id => id, :name => name}
+    end
+
+    return vms
+  end
+
+end
+end
+end

lib/lab/controller/remote_workstation_controller.rb

@@ -0,0 +1,22 @@
+module Lab
+module Controllers
+module RemoteWorkstationController
+
+  def self.running_list(user, host)
+    user.gsub!(/(\W)*/, '')
+    host.gsub!(/(\W)*/, '')
+
+    remote_cmd = "ssh #{user}@#{host} \"vmrun list nogui\""
+    vm_list = `#{remote_cmd}`.split("\n")
+    vm_list.shift
+
+    return vm_list
+  end
+
+  def self.dir_list(basepath=nil)
+    vm_list = Find.find(basepath).select { |f| f =~ /\.vmx$/ }
+    return vm_list
+  end
+end
+end
+end

lib/lab/controller/virtualbox_controller.rb

@@ -0,0 +1,25 @@
+module Lab
+module Controllers
+module VirtualBoxController
+
+  def self.running_list
+    vm_names_and_uuids = `VBoxManage list runningvms`
+    return vm_names_and_uuids.scan(/\"(.*)\" {.*}/).flatten
+  end
+
+  def self.config_list
+    vm_names_and_uuids = `VBoxManage list vms`
+    return vm_names_and_uuids.scan(/\"(.*)\" {.*}/).flatten
+  end
+
+  def self.config_list_uuid
+    vm_names_and_uuids = `VBoxManage list vms`
+    return vm_names_and_uuids.scan(/\".*\" {(.*)}/).flatten
+  end
+
+  def self.dir_list(basepath=nil)
+    vm_list = Find.find(basepath).select { |f| f =~ /\.xml$/ }
+  end
+end
+end
+end