Skip to content
This repository has been archived by the owner on Jan 25, 2023. It is now read-only.

Latest commit

 

History

History

run-nomad

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
run-nomad
run-nomad
 
 

README.md

Nomad Run Script

This folder contains a script for configuring and running Nomad on an AWS server. This script has been tested on the following operating systems:

  • Ubuntu 16.04
  • Ubuntu 18.04
  • Amazon Linux 2

There is a good chance it will work on other flavors of Debian, CentOS, and RHEL as well.

Quick start

This script assumes you installed it, plus all of its dependencies (including Nomad itself), using the install-nomad module. The default install path is /opt/nomad/bin, so to start Nomad in server mode, you run:

/opt/nomad/bin/run-nomad --server --num-servers 3

To start Nomad in client mode, you run:

/opt/nomad/bin/run-nomad --client

This will:

  1. Generate a Nomad configuration file called default.hcl in the Nomad config dir (default: /opt/nomad/config). See Nomad configuration for details on what this configuration file will contain and how to override it with your own configuration.

  2. Generate a systemd configuration file called nomad.service in the systemd config dir (default: /etc/supervisor/conf.d) with a command that will run Nomad:
    nomad agent -config=/opt/nomad/config -data-dir=/opt/nomad/data.

  3. Tell systemd to load the new configuration file, thereby starting Nomad.

We recommend using the run-nomad command as part of User Data, so that it executes when the EC2 Instance is first booting. If you are running Consul on the same server, make sure to use this script after Consul has booted. After running run-nomad on that initial boot, the systemd configuration will automatically restart Nomad if it crashes or the EC2 instance reboots.

Note that systemd logs to its own journal by default. To view the Nomad logs, run journalctl -u nomad.service. To change the log output location, you can specify the StandardOutput and StandardError options by using the --systemd-stdout and --systemd-stderr options. See the systemd.exec man pages for available options, but note that the file:path option requires systemd version >= 236, which is not provided in the base Ubuntu 16.04 and Amazon Linux 2 images.

See the nomad-consul-colocated-cluster example and [nomad-consul-separate-cluster example](https://github.com/hashicorp/terraform-aws-nomad/tree/master/examples/nomad-consul-separate-cluster example) for fully-working sample code.

Command line Arguments

The run-nomad script accepts the following arguments:

  • server (optional): If set, run in server mode. At least one of --server or --client must be set.
  • client (optional): If set, run in client mode. At least one of --server or --client must be set.
  • num-servers (optional): The number of servers to expect in the Nomad cluster. Required if --server is set.
  • config-dir (optional): The path to the Nomad config folder. Default is to take the absolute path of ../config, relative to the run-nomad script itself.
  • data-dir (optional): The path to the Nomad config folder. Default is to take the absolute path of ../data, relative to the run-nomad script itself.
  • systemd-stdout (optional): The StandardOutput option of the systemd unit. If not specified, it will use systemd's default (journal).
  • systemd-stderr (optional): The StandardError option of the systemd unit. If not specified, it will use systemd's default (inherit).
  • user (optional): The user to run Nomad as. Default is to use the owner of config-dir.
  • use-sudo (optional): Nomad clients make use of operating system primitives for resource isolation that require elevated (root) permissions (see the docs for more info). If you set this flag, Nomad will run with root-level privileges. If you don't, it'll still work, but certain task drivers will not be available. By default, this flag is enabled if --client is set and disabled if --server is set (server nodes don't need root-level privileges).
  • skip-nomad-config: If this flag is set, don't generate a Nomad configuration file. This is useful if you have a custom configuration file and don't want to use any of of the default settings from run-nomad.

Example:

/opt/nomad/bin/run-nomad --server --num-servers 3

Nomad configuration

run-nomad generates a configuration file for Nomad called default.hcl that tries to figure out reasonable defaults for a Nomad cluster in AWS. Check out the Nomad Configuration Files documentation for what configuration settings are available.

Default configuration

run-nomad sets the following configuration values by default:

  • advertise: All the advertise addresses are set to the Instance's private IP address, as fetched from
    Metadata.

  • bind_addr: Set to 0.0.0.0.

  • client: This config is only set of --client is set.

  • consul: By default, set the Consul address to 127.0.0.1:8500, with the assumption that the Consul agent is running on the same server.

  • datacenter: Set to the current availability zone, as fetched from Metadata.

  • name: Set to the instance id, as fetched from Metadata.

  • region: Set to the current AWS region, as fetched from Metadata.

  • server: This config is only set if --server is set.

Overriding the configuration

To override the default configuration, simply put your own configuration file in the Nomad config folder (default: /opt/nomad/config), but with a name that comes later in the alphabet than default.hcl (e.g. my-custom-config.hcl). Nomad will load all the .hcl configuration files in the config dir and merge them together in alphabetical order, so that settings in files that come later in the alphabet will override the earlier ones.

For example, to override the default name setting, you could create a file called tags.hcl with the contents:

name = "my-custom-name"

If you want to override all the default settings, you can tell run-nomad not to generate a default config file at all using the --skip-nomad-config flag:

/opt/nomad/bin/run-nomad --server --num-servers 3 --skip-nomad-config

How do you handle encryption?

Nomad can encrypt all of its network traffic (see the encryption docs for details), but by default, encryption is not enabled in this Module. To enable encryption, you need to do the following:

  1. Gossip encryption: provide an encryption key
  2. RPC encryption: provide TLS certificates
  3. Consul encryption

Gossip encryption: provide an encryption key

To enable Gossip encryption, you need to provide a 16-byte, Base64-encoded encryption key, which you can generate using the nomad keygen command. You can put the key in a Nomad configuration file (e.g. encryption.hcl) in the Nomad config dir (default location: /opt/nomad/config):

server {
  encrypt = "cg8StVXbQJ0gPvMd9o7yrg=="
}

RPC encryption: provide TLS certificates

To enable RPC encryption, you need to provide the paths to the CA and signing keys (here is a tutorial on generating these keys). You can specify these paths in a Nomad configuration file (e.g. encryption.hcl) in the Nomad config dir (default location: /opt/nomad/config):

tls {
  # Enable encryption on incoming HTTP and RPC endpoints
  http = true
  rpc  = true

  # Verify server hostname for outgoing TLS connections
  verify_server_hostname = true

  # Specify the CA and signing key paths
  ca_file   = "/opt/nomad/tls/certs/ca-bundle.crt",
  cert_file = "/opt/nomad/tls/certs/my.crt",
  key_file  = "/opt/nomad/tls/private/my.key"
}

Consul encryption

Note that Nomad relies on Consul, and enabling encryption for Consul requires a separate process. Check out the official Consul encryption docs and the Consul AWS Module How do you handle encryption docs for more info.