Add doc for adding execution node

docs/execution_nodes.md

@@ -0,0 +1,78 @@
+# Adding execution nodes to AWX
+
+Stand-alone execution nodes can be added to run alongside the Kubernetes deployment of AWX. These machines will not be a part of the AWX Kubernetes cluster. The control nodes running in the cluster will connect and submit work to these machines via Receptor. The machines be registered in AWX as type "execution" instances, meaning they will only be used to run AWX Jobs (i.e. they will not dispatch work or handle web requests as control nodes do).
+
+Below is an example of a single AWX pod connecting to two different execution nodes. For each execution node, the awx-ee container makes an outbound TCP connection to the machine via Receptor.
+
+```
+                                 AWX POD
+                             ┌──────────────┐
+                             │              │
+                             │ ┌──────────┐ │
+┌─────────────────┐          │ │ awx-task │ │
+│ execution node 1│◄────┐    │ ├──────────┤ │
+├─────────────────┤     ├────┼─┤ awx-ee   │ │
+│ execution node 2│◄────┘    │ ├──────────┤ │
+└─────────────────┘ Receptor │ │ awx-web  │ │
+                      TCP    │ └──────────┘ │
+                     Peers   │              │
+                             └──────────────┘
+```
+
+Note, if the AWX deployment is scaled up, the new AWX pod will also make TCP connections to each execution node.
+
+
+## Overview
+Adding an execution instance involves a handful of steps:
+
+1. [Start a machine that is accessible from the k8s cluster (Red Hat family of operating systems are supported)](#start-machine)
+2. [Create a new AWX Instance with `hostname` being the IP or DNS name of your remote machine.](#create-instance-in-awx)
+3. [Download the install bundle for this newly created instance.](#download-the-install-bundle)
+4. [Run the install bundle playbook against your remote machine.](#run-the-install-bundle-playbook)
+5. [Wait for the instance to report a Ready state. Now jobs can run on that instance.](#wait-for-instance-to-be-ready)
+
+
+### Start machine
+
+Bring a machine online with a compatible Red Hat family OS (e.g. RHEL 8 and 9). This machines needs a static IP, or a resolvable DNS hostname that the AWX cluster can access. The machine will also need an available open port to establish inbound TCP connections on (default is 27199).
+
+In general the more CPU cores and memory the machine has, the more jobs that can be scheduled to run on that machine at once. See https://docs.ansible.com/automation-controller/4.2.1/html/userguide/jobs.html#at-capacity-determination-and-job-impact for more information on capacity.
+
+
+### Create instance in AWX
+
+Use the Instance page or `api/v2/instances` endpoint to add a new instance.
+- `hostname` ("Name" in UI) is the IP address or DNS name of your machine.
+- `node_type` is "execution"
+- `node_state` is "installed"
+- `listener_port` is an open port on the remote machine used to establish inbound TCP connections. Defaults to 27199.
+
+
+### Download the install bundle
+
+On the Instance Details page, click Install Bundle and save the tar.gz file to your local computer and extract contents. Alternatively, make a GET request to `api/v2/instances/{id}/install_bundle` and save the binary output to a tar.gz file.
+
+
+### Run the install bundle playbook
+
+In order for AWX to make proper TCP connections to the remote machine, a few files need to in place. These include TLS certificates and keys, a certificate authority, and a proper Receptor configuration file. To facilitate that these files will be in the right location on the remote machine, the install bundle includes an install_receptor.yml playbook.
+
+The playbook requires the Receptor collection which can be obtained via
+
+`ansible-galaxy collection install -r requirements.yml`
+
+Modify `inventory.yml`. Set the `ansible_user` and any other ansible variables that may be needed to run playbooks against the remote machine.
+
+`ansible-playbook -i inventory.yml install_receptor.py` to start installing Receptor on the remote machine.
+
+Note, the playbook will enable the [Copr ansible-awx/receptor repository](https://copr.fedorainfracloud.org/coprs/ansible-awx/receptor/) so that Receptor can be installed.
+
+
+### Wait for instance to be Ready
+
+Wait a few minutes for the periodic AWX task to do a health check against the new instance. The instances endpoint or page should report "Ready" status for the instance. If so, jobs are now ready to run on this machine!
+
+
+## Removing instances
+
+You can remove an instance by clicking "Remove" in the Instances page, or by setting the instance `node_state` to "deprovisioning" via the API.