Skip to content

Commit 8d28ba4

Browse files
wmudgewmudge
authored
Add Molecule testing, including rdbms_server (#197)
* Add molecule testing for cloudera.exe.rdbms_server to collection, including pytest integration * Add deprecation warning to nested role (cloudera.exe.rdbms.server) * Suppress 'crypt' deprecation warning * Add skeleton tox-ansible configuration * Update testing instructions for pytest-ansible and Molecule usage * Add default Molecule scenario * Add fully qualified RHEL UBI images * Add psycopg2-binary installation on RHEL hosts * Update PostgreSQL repositories to handle arm64 architectures * Remove explicit architecture reference in apt repository for Postgresql RHEL installs Signed-off-by: Webster Mudge <wmudge@cloudera.com>
1 parent dd51746 commit 8d28ba4

File tree

32 files changed

+1145
-30
lines changed

32 files changed

+1145
-30
lines changed

‎.config/molecule/config.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright 2024 Cloudera, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
provisioner:
16+
name: ansible
17+
config_options:
18+
defaults:
19+
collections_path: ${ANSIBLE_COLLECTIONS_PATH}

‎TESTING.md‎

Lines changed: 183 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,195 @@
11
# Testing cloudera.exe
22

3-
The collection uses `ansible-test` for unit and integration testing.
3+
The collection is migrating from `ansible-test` to `pytest` and `molecule` for unit and integration testing. In addition, we use `pre-commit` to handle linting and formatting both for `git` commits and for various Github Action workflows in the repository.
4+
5+
## Setup
6+
7+
To set up a development and test environment for the collection, you need to:
8+
9+
1. Set up the Ansible Collection and Role paths
10+
1. Install Ansible and the Python dependencies
11+
1. Install the collection and its dependencies
12+
1. Configure the PYTHONPATH to use the correct location of the collections code
13+
1. Install the Molecule driver dependencies
14+
15+
### Ansible Collection and Role Paths
16+
17+
You have to install your Ansible collections, both the collection under test and its dependencies, into the `ansible_collections/<namespace>/<name>` folder structure. For the collection under test, run the following _in the parent directory of your choosing_:
18+
19+
```bash
20+
git clone https://github.com/cloudera-labs/cloudera.exe.git ansible_collections/cloudera/exe
21+
```
22+
23+
Then create the `roles` directory in the _parent directory_:
24+
25+
```bash
26+
mkdir roles
27+
```
28+
29+
Lastly, set the Ansible [COLLECTION](https://docs.ansible.com/ansible/latest/reference_appendices/config.html#envvar-ANSIBLE_COLLECTIONS_PATH) and [ROLE](https://docs.ansible.com/ansible/latest/reference_appendices/config.html#envvar-ANSIBLE_ROLES_PATH) configurations for these two locations:
30+
31+
```bash
32+
export ANSIBLE_COLLECTIONS_PATH=$(pwd)
33+
export ANSIBLE_ROLES_PATH="$(pwd)/roles"
34+
```
35+
36+
### Set the PYTHONPATH
37+
38+
Include the `ANSIBLE_COLLECTIONS_PATH` variable to the `PYTHONPATH` to allow module imports.
39+
40+
```bash
41+
export PYTHONPATH="${ANSIBLE_COLLECTIONS_PATH}":"${PYTHONPATH}"
42+
```
43+
44+
### Ansible
45+
46+
Set up a development `virtualenv` and install `ansible-core~=2.16.0` and `ansible-navigator`.
47+
48+
```bash
49+
pip install ansible-core~=2.16.0 ansible-navigator
50+
```
51+
52+
> [!warning]
53+
> Installing `>=2.17` will require that the target hosts run Python 3.7. This requirement extends to RHEL 8.x and its `platform-python`, which means that `2.17` will not work on these platforms.
54+
55+
### Python Dependencies
56+
57+
Install the development and collection Python requirements from the `requirements-dev.txt` and `requirements.txt` files respectively in the project root.
58+
59+
```bash
60+
pip install -r ansible_collections/cloudera/exe/requirements.txt
61+
pip install -r ansible_collections/cloudera/exe/requirements-dev.txt
62+
```
63+
64+
### Collection Dependencies
65+
66+
You also need to install the collection's dependencies and install them into the `ANSIBLE_COLLECTIONS_PATH`:
67+
68+
```bash
69+
ansible-galaxy collection install -r ansible_collections/cloudera/exe/requirements.yml -p "${ANSIBLE_COLLECTIONS_PATH}"
70+
```
71+
72+
And install any role dependencies as well into the `ANSIBLE_ROLES_PATH`:
73+
74+
```bash
75+
ansible-galaxy role install -r ansible_collections/cloudera/exe/requirements.yml -p "${ANSIBLE_ROLES_PATH}"
76+
```
77+
78+
If the collection has any system requirements, run `bindep` on its requirements file:
479

580
```bash
6-
# Sanity tests
7-
ansible-test sanity --docker --python 3.9
81+
bindep -f ansible_collections/cloudera/exe/bindep.txt
82+
```
83+
84+
### Molecule
85+
86+
Running the `molecule` tests requires `podman` as the container engine, so you will need to install that service on your test machine. Read more about [Podman](https://podman.io/) or [Podman Desktop](https://podman-desktop.io/).
87+
88+
## Testing
89+
90+
You can either run standalone `molecule`, for roles and more advanced integration testing, or `pytest`. The latter is set up to run any and all tests, including `molecule` scenarios.
91+
92+
### Running standalone `molecule` tests
93+
94+
Currently, `molecule` scenarios are located in the `extensions/molecule` directory of the collection. To run a scenario, change to `extensions` as your current working directory and then run `molecule`. For example:
95+
96+
| Command | Description |
97+
| --- | --- |
98+
| `molecule test -s rdbms_server_postgresql_14_tls` | Execute the full test lifecyle for the PostgreSQL 14 server role with TLS |
99+
| `molecule create -s rdbms_server_postgresql_14_tls` | Create the `platforms`, i.e. the inventory, that are the target hosts of the role testing |
100+
| `molecule prepare -s rdbms_server_postgresql_14_tls` | Prep the target hosts for testing the roles |
101+
| `molecule converge -s rdbms_server_postgresql_14_tls` | Run the testing playbook, i.e. converge the test code, on the target hosts |
102+
| `molecule side-effect -s rdbms_server_postgresql_14_tls` | Prep the target hosts, post-`converge`, for any additional setup prior to verification or idempotency testing |
103+
| `molecule verify -s rdbms_server_postgresql_14_tls` | Verify the target hosts |
104+
| `molecule cleanup -s rdbms_server_postgresql_14_tls` | Clean up any resources, for example, temporary files created on the controller |
105+
| `molecule destroy -s rdbms_server_postgresql_14_tls` | Destroy the `platform` hosts |
106+
107+
You can limit testing to a `platform` within a scenario by using the `-p/--platform-name` parameter (or via the `MOLECULE_PLATFORM_NAME` environment variable):
108+
109+
```bash
110+
molecule test -s rdbms_server_postgresql_14_tls -p rhel9.4
111+
```
112+
113+
To stop tests from destroying the platforms after encountering an error (or at all, even on a successful test), pass the `--destroy=never` parameter:
114+
115+
```bash
116+
molecule test -s rdbms_server_postgresql_14_tls -p rhel9.4 --destroy=never
117+
```
118+
119+
You can log into a running platform via the `login` subcommand and the `-h/--host` parameter:
120+
121+
```bash
122+
molecule login -s rdbms_server_postgresql_14_tls -h rhel9.4
123+
```
124+
125+
As well as pass extra parameters to the underlying playbook (`converge` command only!):
8126

9-
# Unit tests
10-
ansible-test units --docker --python 3.9
127+
```bash
128+
molecule converge -s rdbms_server_postgresql_14_tls -- -vvv -t tls_config
129+
```
130+
131+
### Running `pytest` tests
132+
133+
We use the `ansible-pytest` plugin to run unit and integration tests for the collection.
134+
135+
To see what tests (unit and integration) are available, run the following from the `ANSIBLE_COLLECTIONS_PATH` directory:
136+
137+
```bash
138+
pytest ansible_collections/cloudera/exe --collect-only
139+
```
140+
141+
You should see something like:
142+
143+
```
144+
platform darwin -- Python 3.12.4, pytest-8.3.3, pluggy-1.5.0
145+
ansible: 2.16.11
146+
rootdir: /Users/wmudge/Devel/ansible_collections/cloudera/exe
147+
configfile: pyproject.toml
148+
testpaths: tests
149+
plugins: ansible-24.9.0, xdist-3.6.1
150+
collected 8 items
11151
12-
# Integration tests
13-
ansible-test integration --docker
152+
<Dir exe>
153+
<Dir tests>
154+
<Dir integration>
155+
<Module test_molecule_integration.py>
156+
<Function test_integration[extensions-rdbms_server_postgresql_14_tls]>
157+
<Function test_integration[extensions-rdbms_server_postgresql_default]>
158+
<Function test_integration[extensions-rdbms_server_postgresql_14]>
159+
<Function test_integration[platform-default]>
160+
<Function test_integration[platform-level0]>
161+
<Function test_integration[runtime-default]>
162+
<Function test_integration[runtime-level0]>
163+
<Package unit>
164+
<Package plugins>
165+
<Package filter>
166+
<Module test_core_exe.py>
167+
<UnitTestCase TestFilterModule>
168+
<TestCaseFunction test_combine_onto>
14169
```
15170

16-
To run the _integration_ tests, you first need to have a running virtual environment configured with Ansible (`core`) and the required collections. When you run `ansible-test`, the program will bootstrap the [requirements|tests/integration/requirements.txt] in the Docker container and mount the `ANSIBLE_COLLECTION_PATHS`.
171+
To run a selected test, execute with a regex:
17172

18173
```bash
19-
# In your favorite VENV...
20-
pip3 install ansible-core
21-
ansible-galaxy collection install -p collections -r galaxy.yml
22-
export ANSIBLE_COLLECTION_PATHS="$(pwd)/collections"
174+
pytest ansible_collections/cloudera/exe -k "postgresql_14_tls"
23175
```
24176

25-
You also need to provide AWS credentials and test configuration in the `integration_config.yml` file. This file is *not* included in the project, as it will contain sensitive data, but there is a template -- `integration_config.yml.template` -- that you can copy and update as needed.
177+
To run a Molecule scenario on a selected platform, i.e. target host, set the platform via the environment variable:
178+
179+
```bash
180+
MOLECULE_PLATFORM_NAME="rhel9.4" pytest ansible_collections/cloudera/exe -k "postgresql_14_tls"
181+
```
182+
183+
> [!warning]
184+
> If you run `pytest` in the root of the collection, `pytest` will copies the current collection into the required Ansible collection path structure within the working directory. That is, running `pytest` at the root of the **collection** creates a `collection/ansible_collections/<namespace>/<name>` **within** the collection.
185+
> Thus, our recommendation is that you can run `pytest` in at the root of the **Ansible collections path**. That is, run `pytest ansible_collections/<namespace>/<name> ...` so that `pytest` doesn't have to bootstrap the collections path.
186+
187+
## Linting and Commits
188+
189+
The `pre-commit` Python application is used to manage linting and other housekeeping functions. The application is installed as a `git` hook and as a Github workflow check.
190+
191+
Commits and pull requests will fail if your contributions do not pass the `pre-commit` checks. You can check your work-in-progress code by running the following:
192+
193+
```bash
194+
pre-commit run -a
195+
```

‎extensions/molecule/default/Dockerfile-rhel.j2‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Copyright 2024 Cloudera, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Derived from molecule-plugins[podman]
16+
17+
{% if item.registry is defined %}
18+
FROM {{ item.registry.url }}/{{ item.image }}
19+
{% else %}
20+
FROM {{ item.image }}
21+
{% endif %}
22+
23+
{% if item.env is defined %}
24+
{% for var, value in item.env.items() %}
25+
{% if value %}
26+
ENV {{ var }} {{ value }}
27+
{% endif %}
28+
{% endfor %}
29+
{% endif %}
30+
31+
RUN dnf makecache && dnf --assumeyes install \
32+
python3 \
33+
python3-devel \
34+
python3-dnf \
35+
sudo \
36+
bash \
37+
iproute \
38+
&& dnf clean all
39+
40+
CMD ["/sbin/init"]

‎extensions/molecule/default/Dockerfile-ubuntu.j2‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Copyright 2024 Cloudera, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Derived from molecule-plugins[podman]
16+
17+
{% if item.registry is defined %}
18+
FROM {{ item.registry.url }}/{{ item.image }}
19+
{% else %}
20+
FROM {{ item.image }}
21+
{% endif %}
22+
23+
{% if item.env is defined %}
24+
{% for var, value in item.env.items() %}
25+
{% if value %}
26+
ENV {{ var }} {{ value }}
27+
{% endif %}
28+
{% endfor %}
29+
{% endif %}
30+
31+
ARG DEBIAN_FRONTEND=noninteractive
32+
33+
RUN apt-get update && apt-get install -y --no-install-recommends \
34+
python3 \
35+
python3-pip \
36+
python3-apt \
37+
python3-dev \
38+
sudo \
39+
bash \
40+
gpgv2 \
41+
systemd \
42+
systemd-cron \
43+
xmlsec1 \
44+
libxmlsec1-openssl \
45+
libpq-dev \
46+
&& apt-get clean \
47+
&& rm -Rf /usr/share/doc && rm -Rf /usr/share/man \
48+
&& rm -rf /var/lib/apt/lists/*
49+
50+
CMD ["/lib/systemd/systemd"]

‎extensions/molecule/default/converge.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Copyright 2024 Cloudera, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
- name: Default test
16+
hosts: all
17+
gather_facts: yes
18+
tasks:
19+
- name: Heartbeat
20+
ansible.builtin.ping:

‎extensions/molecule/default/molecule.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Copyright 2024 Cloudera, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
driver:
16+
name: podman
17+
dependency:
18+
name: galaxy
19+
options:
20+
requirements-file: "${MOLECULE_SCENARIO_DIRECTORY}/requirements.yml"
21+
role-file: "${MOLECULE_SCENARIO_DIRECTORY}/requirements.yml"
22+
platforms:
23+
- name: rhel9.4
24+
image: registry.access.redhat.com/ubi9/ubi:9.4
25+
dockerfile: Dockerfile-rhel.j2
26+
command: /sbin/init
27+
- name: rhel8.10
28+
image: registry.access.redhat.com/ubi8/ubi:8.10
29+
dockerfile: Dockerfile-rhel.j2
30+
command: /sbin/init
31+
- name: ubuntu22.04
32+
image: ubuntu:22.04
33+
dockerfile: Dockerfile-ubuntu.j2
34+
command: /lib/systemd/systemd
35+
- name: ubuntu20.04
36+
image: ubuntu:20.04
37+
dockerfile: Dockerfile-ubuntu.j2
38+
command: /lib/systemd/systemd

‎extensions/molecule/default/requirements.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Copyright 2024 Cloudera, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
collections:
16+
- containers.podman
17+
- ansible.posix

0 commit comments

Comments
 (0)