Jetpants supports a global configuration file at /etc/jetpants.yaml, as well as per-user configuration files at ~/.jetpants.yaml.
At least one of these files must exist for Jetpants to function properly.
If both exist, the user configuration will be merged on top of the global configuration, this is a “deep” merge. So if the user a “plugins” section this will be combined with the global “plugins” section.
For an example global configuration file, please see the included etc/jetpants.yaml.sample file.
- max_concurrency
-
Maximum threads to use in import/export operations, or equivalently the maximum connection pool size per database host. You will need to tune this to your specific database hardware, especially number of CPU cores and number/type of disks (default: 20)
- standby_slaves_per_pool
-
Minimum number of standby slaves to keep in every pool (default: 2)
- mysql_schema
-
database name (mandatory)
- mysql_app_user
-
mysql user that your application uses (mandatory)
- mysql_app_password
-
mysql password that your application uses (mandatory)
- mysql_repl_user
-
mysql user for replication (mandatory)
- mysql_repl_password
-
mysql password for replication (mandatory)
- mysql_root_password
-
mysql root password (default: false, indicating that Jetpants should use /root/.my.cnf instead)
- mysql_grant_ips
-
mysql user manipulations are applied to these IPs (array; mandatory)
- mysql_grant_privs
-
mysql user manipulations grant this set of privileges by default (array; default: [‘ALL’])
- compress_with
-
command line to perform compression during large file copy operations; see below (default: false)
- decompress_with
-
command line to perform decompression during large file copy operations; see below (default: false)
- export_location
-
directory to use for data dumping (default: ‘/tmp’)
- verify_replication
-
raise exception if the actual replication topology differs from Jetpants’ understanding of it (ie, disagreement between asset tracker and probed state), or if MySQL’s two replication threads are in different states (one running and the other stopped) on a DB node. (default: true. master promotion tool ignores this, since the demoted master may legitimately be dead/offline)
- private_interface
-
name of private interface on your servers, such as eth0 or bond0. Not used by any core Jetpants commands, but can be useful in plugings that wrap tcpdump, or code calling the Jetpants::Host.local method. (default: ‘bond0’)
- plugins
-
hash of plugin name => arbitrary plugin data, usually a nested hash of settings (default: {})
- ssh_keys
-
array of SSH private key file locations, if not using standard id_dsa or id_rsa. Passed directly to Net::SSH.start’s :keys parameter (default: nil)
- sharded_tables
-
array of name => {sharding_key=>X, chunks=>Y} hashes, describing all tables on shards. Required by shard split/rebuild processes (default: [])
Jetpants has the ability to use compression during large file copy operations, which are performed by commands “jetpants clone_slave” and “jetpants shard_split”. Compression is disabled by default in Jetpants unless you specify a compression program to use via the compress_with and decompress_with config options. It is highly recommended that you do so, in order to speed up these operations when working with large data sets.
The command lines that you specify should accept input from STDIN and supply output to STDOUT, because they will be used in the middle of a series of piped commands. The binary specified should be in root’s PATH on all database nodes. We recommend use of a parallel compression tool, to take advantage of multiple cores.
You will need to do some profiling to determine the best tool to use for your hardware and data set; there’s no universal best choice of compression algorithm or settings.
Some example values of these parameters are as follows:
compress_with: false decompress_with: false
pigz is an open-source parallel gzip tool by Mark Adler. It is available as a package in several Linux distros. It performs well, but is very CPU intensive. More information: zlib.net/pigz/
compress_with: pigz decompress_with: pigz -d
qpress is a multi-threaded portable file archiver using QuickLZ. A prebuilt package is not available for most Linux distros due to licensing reasons, but a binary can be downloaded from www.quicklz.com/. It performs extremely well, especially once tuned.
In order to read from STDIN and write to STDOUT, use qpress -io. In this case qpress still requires a filename during compression, even though it is unused. Decompression does not have the same requirement.
The example below uses 4 threads and a block size of 32768KB.
compress_with: qpress -ioT4K32768 dummyfilename decompress_with: qpress -dioT4
lzop is a less CPU-intensive compressor. lzop is still single-threaded in v1.x, so its performance may not be ideal for the Jetpants use-case. Multithreading is planned for v2.x. More information: www.lzop.org/
compress_with: lzop decompress_with: lzop -d