Skip to content

A Leiningen plugin for running tasks against a live Hadoop cluster

License

Notifications You must be signed in to change notification settings

llasram/lein-hadoop-cluster

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

lein-hadoop-cluster

A Leiningen plugin for running tasks against a live Hadoop cluster. Adds a profile named plugin.lein-hadoop-cluster/default which includes the correct class path etc for the currently-configured Hadoop cluster.

Usage

Put the following dependency into the :plugins vector of your project.clj:

[lein-hadoop-cluster "0.1.5"]

Due to technomancy/leiningen#1712, this currently will not work as a profile-level plugin e.g. in your :user profile. It must be a top-level plugin in each project where you intend to use the plugin.

With the plugin added to the project, you can run tasks under the plugin.lein-hadoop-cluster/default profile, or otherwise use the profile to access the live cluster task/JVM configuration. For example:

$ lein with-profile plugin.lein-hadoop-cluster/default trampoline repl

Internally, the plugin defines and makes use of two Leiningen profiles:

  • plugin.lein-hadoop-cluster/default – Intended as a replacement for the top-level default. By inncludes the profiles system, base, dev, user, and plugin.lein-hadoop-cluster/system.
  • plugin.lein-hadoop-cluster/system – Additive profile containing the actual configuration necessary to access the system Hadoop installation.

Additionally, the plugin expects that project files will specify all their Hadoop-provided dependencies via the provided profile. Placing them in e.g. dev instead will result in them appearing on the cluster profile classpaths before the system Hadoop, masking your live cluster configuration.

License

Copyright © 2014-2015 Marshall Bockrath-Vandegrift

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.

About

A Leiningen plugin for running tasks against a live Hadoop cluster

Resources

License

Stars

Watchers

Forks

Packages

No packages published