Skip to content

Commit 923b096

Browse files
maxrobertmaxrobert
authored
Merge pull request #17 from jmcgee-geon/develop-2.2
Add container orchestration capabilities
2 parents 7f72db1 + 173f006 commit 923b096

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

72 files changed

+4243
-434
lines changed

‎docs/container-orchestration/README.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# REDHAWK Container Orchestration
2+
REDHAWK offers a standardized methodology for running waveforms on container orchestration technologies by means of an extensible plugin class. These capabilities have been integrated into core-framework through modifications to the Application Factory and Sandbox but in a manner that emphaised backwards compatibility; it is still possible to use the standard Domain Manager and GPP combo to run native Executable and SharedLibrary type components on your REDHAWK systems.
3+
4+
Each cluster technology has its own unique ways to launch, configure, and delete containers, so the plugin system allows users to develop their own Sandbox and Application Factory plugins that Domain Manager dynamically loads at run-time to spawn waveforms on a cluster according a global cluster configuration file called `cluster.cfg` that controls the parameters passed to each plugin.
5+
6+
*NOTE* These additions have *not* been reconciled with REDHAWK IDE. Viewing Components with spd.xml files modified in support of using these features will generate visual errors in RHIDE.
7+
8+
# General Installation Dependencies
9+
10+
1. Install the dependencies for REDHAWK with the [REDHAWK Manual](https://redhawksdr.org/2.2.8/manual/appendices/dependencies/)
11+
2. Yum install the `yaml-cpp-devel` package for the plugins' yaml generation
12+
3. Docker version 19.03.12 is required for building REDHAWK Component docker images
13+
4. Download and install [docker-redhawk](https://github.com/Geontech/docker-redhawk.git) tag 2.2.8-0 or [docker-redhawk-ubuntu](https://github.com/Geontech/docker-redhawk-ubuntu.git) branch develop-2.2.1 (Ubuntu is only required for GNURadio components)
14+
5. The following must be present in your environment (have network connectivity to your system running REDHAWK and proper configuration) in order to use RH components in clusters:
15+
* A cluster running the desired orchestration technology
16+
* The matching RH cluster plugin installed
17+
* Your networking, firewalls and DNS setup to communicate with your cluster's control plane/nodes/API
18+
* Your local system with REDHAWK installed must have its `cluster.cfg` file set correctly and the plugin's approprriate environment variables set
19+
20+
For plugin specific instructions, please see that plugin's corresponding documentation.
21+
22+
# Plugin Execution Path
23+
The Application Factory loops through the list of components in the waveform making note of the component's code type. If the code type is set to "Container", then the spd parsing mechanism checks the component's entrypoint. The entrypoint element has been extended to identify the suffix of a docker image path. Consider the example below pulled from SigGen's spd.xml file.
24+
```
25+
<code type="Container">
26+
<localfile name="cpp/SigGen"/>
27+
<entrypoint>cpp/SigGen::rh.siggen</entrypoint>
28+
</code>
29+
```
30+
In this entrypoint element, everything after the "::" is regarded as the desired image name suffix. The "::" is a delimiter. The prefix of the image is found in the `cluster.cfg` file.
31+
32+
If all components in the waveform are of Container code type, then no GPP is required to run the waveform. Waveforms can be ran in a "hybrid" manner in which components with Executable or SharedLibrary code types will run on the native REDHAWK system while those with Container code type are executed on the cluster dictated by the installed plugin and `cluster.cfg` file. Hybrid waveforms, or any waveform that uses Executable or SharedLibrary code types still required a GPP to run (this is for backwards compatibility).
33+
34+
# Building and Installing core-framework from Source
35+
```bash
36+
$ cd /core-framework/redhawk/src
37+
$ ./build.sh
38+
$ sudo make install
39+
```
40+
## Building and Installing Plugins
41+
No plugin is installed by default. Three plugins are currently included with REDHAWK:
42+
1. [Docker](plugin-docker.md)
43+
2. [DockerSwarm](plugin-dockerswarm.md)
44+
3. [EksKube](plugin-ekskube.md)
45+
46+
## The cluster.cfg file
47+
The `$OSSIEHOME/cluster.cfg` file contains sections named according to the cluster orchestration plugin technology (EKS, Docker Swarm, etc). Each plugin-specific section contains variables (key value pairs) used by the specific plugin. You should ensure the values set in this file are correct prior to running a waveform.
48+
49+
#### Installing the Cluster Configuration File
50+
```bash
51+
$ cd ./core-framework/redhawk/src/base/cfg
52+
$ sudo -E ./build.py --cluster <Desired_Plugin_Name> <Arguments to set variables in cluster.cfg>
53+
```
54+
55+
This will render the template cluster.cfg file into your $OSSIEHOME/cluster.cfg file. The plugin name argument you specify sets the plugin's top section which controls will plugin REDHAWK will use, assuming the plugin is already installed.
56+
57+
Possible values for the plugin name argument are:
58+
1. [EksKube](plugin-ekskube.md)
59+
2. [Docker](plugin-docker.md)
60+
3. [DockerSwarm](plugin-dockerswarm.md)
61+
62+
## Networking
63+
Networking can get complicated quickly because it will vary depending on what you're trying to run in your cluster, what that cluster's networking setup looks like, and your local system's ability to send traffic to that cluster. Guiding rules of thumb are:
64+
* If you are running DomMgr and/or GPP with OmniORB on your local system and only running waveforms on a cluster, those launched components need to be able to initiate connections to omniORB. This boils down to there being a "flat" network between where OmniORB runs and where the waveform runs. NAT will break one party's ability to talk to the other.
65+
* In the scenario where all REDHAWK services (OmniORB, DomMgr, and/or GPP) run inside the cluster alongside the waveform payloads, so long as the containers can network resolve each other (almost always the case barring network security restrictions on the cluster), then there should not be any difficulties with networking.
66+
67+
Please see each plugin's documention for more more network specifics.
68+
69+
## Misc additions
70+
Each plugin behaves differently. Some require specialized networking, other require special credentials, and some might require environment variables. Please consult the specific plugin's documentation to learn more.

‎docs/container-orchestration/plugin-class.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Plugin Class
2+
The parent plugin class can be used to inherit from and form the skeleton of your own plugin. It can be found at `core-framework/src/base/include/ossie/cluster/ClusterManagerResolver.h` The class' public methods are those that your derivative class can overwrite for you orchestration technology's way of handling containers.
3+
4+
# Public Methods
5+
## launchComponent(std::string app_id)
6+
* Launches a component or yaml file of multiple components into the cluster
7+
* @param app_id the redhawk code passes the plugin the application ID (can be used for namespaces for example)
8+
* @return An integer representing a pid. A negative pid will throw an error while a pid 0 and greater will succeed
9+
10+
## pollStatusActive(std::string app_id)
11+
* Polls the component and waits to see that it is active and running (equivalent to REDHAWKs native pid check but for clusters)
12+
* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be active
13+
* @return Boolean where true means the component is active and false means the component is not. REDHAWK prints an error to logs if false
14+
15+
## pollStatusTerminated(std::string app_id)
16+
* Polls the component and waits for terminatation (in clusters this might mean CrashLoopBackoff, Completed, etc...)
17+
* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be terminated
18+
* @return Boolean where true means the component is terminated and false means the component is not. REDHAWK prints an error to logs if false
19+
20+
## deleteComponent(std::string comp_id)
21+
* Deletes a component or multiple components in a yaml file from the namespace
22+
* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be deleted
23+
24+
## isTerminated(std::string app_id)
25+
* One-off check for if a component has terminated (in clusters this might mean CrashLoopBackoff, Completed, etc...)
26+
* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be terminated
27+
* @return true if terminated and false if not. Throws a ComponentTerminated exception if false on start up
28+
29+
## openComponentConfigFile(redhawk::PropertyMap execParameters, std::string entryPoint, std::string image)
30+
Adds a component to the yaml file so that all cluster type components are in the file before launching. This is also the location where non-cluster type deployments would run executables (see DockerResolver.cpp)
31+
32+
`execParameters` The parameters given to the component to be able to execute it. These parameters can instead be baked into the yaml file for the cluster technology to launch itself
33+
(i.e /path/to/executable NAMING_CONTEXT_IOR <nameing_context_ior> PROFILE_NAME <profile_name> NAME_BINDING <name_binding> COMPONENT_IDENTIFIER <component_identifier> DEBUG_LEVEL <debug_level>).
34+
Other params include:
35+
* NIC
36+
* RH::GPP::MODIFIED_CPU_RESERVATION_VALUE
37+
38+
`entryPoint` The path to the executable (i.e /var/redhawk/sdr/dom/components/rh/SigGen/cpp/SigGen)
39+
40+
`image` The image name that was attached to the entrypoint in the spd.xml file (i.e in the spd <\entrypoint>/path/to/executable::image<\entrypoint>).
41+
This is not the fully qualified path to the image. The registry path will instead be found in /usr/local/redhawk/core/cluster.cfg and combined with this `image` parameter to yield the fully qualified image path.
42+
43+
## closeComponentConfigFile(std::string app_id)
44+
* Closes the yaml file that was being written to
45+
* @param app_id the application ID is given so that when the file is saved out it can be unique
46+
47+
48+
49+

‎docs/container-orchestration/plugin-docker.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Docker
2+
The Docker plugin is designed to run REDHAWK waveforms on your local system's Docker installation.
3+
4+
# Building and Installing the Plugin
5+
6+
Both the Application Factory and Sandbox plugins for Docker are installed when core-framework is built and installed from source.
7+
8+
Application Factory
9+
```bash
10+
$ cd core-framework/redhawk/src/base/plugin/clustermgr/clustertype
11+
$ ./build.py Docker
12+
$ ./reconf && ./configure && make && sudo make install
13+
```
14+
Sandbox
15+
```bash
16+
$ cd core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox
17+
$ make FILE=Docker
18+
```
19+
20+
This will compile and install the Application Factory and Sandbox plugins for the user. The plugins are built in a specific location in core-framework (`core-framework/redhawk/src/base/plugin/clustermgr/`and `core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox/`) and are both installed to `/usr/local/redhawk/core/lib`
21+
22+
# Plugin Specifics
23+
## Dependencies
24+
1. Docker installed on your system along with REDHAWK
25+
26+
## The cluster.cfg file
27+
```bash
28+
cd core-framework/redhawk/src/base/cfg
29+
sudo -E ./build.py --cluster Docker --docker_dir <where-to-mount-on-container> --mount_dir <where-to-mount-from>
30+
```
31+
OR
32+
```bash
33+
cd core-framework/redhawk/src/base/cfg
34+
make Docker DOCKER_DIR="<your-docker-dir>" MOUNT_DIR="<your-mount-location>"
35+
```
36+
This will properly set the top section to use the Docker plugin and pass in the assortment of arguments to setup the cluster.cfg file.
37+
38+
## cluster.cfg file variables
39+
The top section of the file should specify that the Docker plugin is desired like so:
40+
```
41+
[CLUSTER]
42+
name = Docker
43+
```
44+
| Variable | Example Value | Description |
45+
|------------|-----------------------|------------------------------------------------------------|
46+
| docker_dir | /mnt/ | Path inside of the docker container to mount mount_dir to |
47+
| mount_dir | /home/bob/myshareddir | Path on the docker host to mount into the docker container |
48+
49+
*NOTE*: These variables are only used by the Sandbox Docker plugin and do not work for the Application Factory plugin.
50+
51+
## Credentials
52+
The Docker plugin needs the following credentials:
53+
1. Your ~/.docker/config.json updated to allow docker to pull your Component images from your desired Registry if the images are not already present on your local system
54+
55+
## Networking
56+
Docker uses a bridged network by default by running the containers with the "--network host" option.
57+
58+
This configuration is ideal for enabling your containers to communicate with running REDHAWK services on the same system.

‎docs/container-orchestration/plugin-dockerswarm.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# DockerSwarm
2+
The DockerSwarm plugin is designed to run REDHAWK waveforms on Docker Swarm clusters.
3+
4+
# Building and Installing the Plugin
5+
6+
Application Factory
7+
```bash
8+
$ cd core-framework/redhawk/src/base/plugin/clustermgr/clustertype
9+
$ sudo ./build.py DockerSwarm
10+
$ ./reconf && ./configure && make && sudo make install
11+
```
12+
Sandbox
13+
```bash
14+
$ cd core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox/clustertype
15+
$ make FILE=DockerSwarm
16+
```
17+
18+
This will compile and install the Application Factory and Sandbox plugins for the user. The plugins are built in a specific location in core-framework (`core-framework/redhawk/src/base/plugin/clustermgr/`and `core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox/`) and are both installed to `/usr/local/redhawk/core/lib`
19+
20+
# Plugin Specifics
21+
## Dependencies
22+
1. None
23+
24+
## The cluster.cfg file
25+
```bash
26+
cd core-framework/redhawk/src/base/cfg
27+
sudo -E ./build.py --cluster DockerSwarm --registry <your-registry> --ssh_key <your-ssh-key> --server_user <your-server-username> --server_ip <your-server-ip-addr>
28+
```
29+
OR
30+
```bash
31+
cd core-framework/redhawk/src/base/cfg
32+
make DockerSwarm REGISTRY="<your-registry>" SSH_KEY="<your-ssh-key>" SERVER_USER="<your-server-username>" SERVER_IP="<your-server-ip-addr>"
33+
```
34+
This will properly set the top section to use the DockerSwarm plugin and pass in the assortment of arguments to setup the cluster.cfg file.
35+
36+
## cluster.cfg file variables
37+
The top section of the file should specify that the DockerSwarm plugin is desired like so:
38+
```
39+
[CLUSTER]
40+
name = DockerSwarm
41+
```
42+
| Variable | Example Value | Description |
43+
|------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
44+
| registry | geontech | This value is concatenated with a "/" and then the image suffix found in the entrypoint of the component's spd.xml file. Shared across all components. |
45+
| tag | latest | The image tag used. Shared across all components. |
46+
| key | ~/.ssh/ssh_key.pem | Path to your SSH key. Must be read-accessible by Domain Manager |
47+
| user | centos | User used to log into node via SSH |
48+
| ip | 10.10.10.10 | IP address or FQDN of the Swarm Master node to connect to via SSH |
49+
| docker_login_cmd | "docker login" | The command ran by the plugin to log into the registry hosting the Component container images |
50+
51+
## Credentials
52+
The DockerSwarm plugin needs the following credentials:
53+
1. A linux user on a Docker Swarm Manager node authorized to SSH into it with a corresponding SSH private key
54+
2. Docker login creds on the Docker Swarm Manager Node
55+
56+
Docker Swarm does not offer a "remote control" capability like K8S does via kubectl, so the Docker Swarm plugin uses SSH to log into a Swarm Manager Node to run/apply the generated docker-compose yaml. For this SSH login to work, a pre-configured user must exist on the targeted Manager node that is allowed to SSH in.
57+
58+
When the plugin logs into the node over SSH, it first runs the command defined in the docker_login_cmd variable in `cluster.cfg`. You can overrite this value to supply your username and password as arguments in the `cluster.cfg` file.
59+
60+
## Networking
61+
Docker Swarm uses virtual overlay networks by default, which isn't ideal for waveforms on the Swarm and REDHAWK services running on a remote system external to the cluster. It is for this reason that the yaml generated for Docker Swarm creates a host network and attached all Component containers to that network:
62+
```yaml
63+
version: '3'
64+
networks:
65+
outside:
66+
external:
67+
name: host
68+
services:
69+
siggen1:
70+
...
71+
networks:
72+
- outside
73+
````
74+
This makes each Component container share the host node's IP address.
75+
76+
77+
This configuration is ideal for running REDHAWK services external to the cluster (on your local REDHAWK system).

0 commit comments

Comments
 (0)