Skip to content

Commit c4f8d30

Browse files
wmudgejimright
wmudge
and
jimright
authored
Add integration targets and rebase onto devel (#1)
* Add integration test for cross-account credentials (#90) * Add integration targets for CDP Environment and general teardown (#91) * Add multiAZ for AWS environment * Add multi-AZ support for FreeIPA at environment creation * Add multi-AZ support for datalake * Add a common testing role for shared variables * Create integration test for CDP Credentials * Add shared integration CDP Environments for downstream tests (single and multi-AZ) * Create shared infrastructure resources teardown * Rename integration target xaccount to cred * Add and update integration targets for CDP Environments (single and multi-AZ) * Add integration targets for CDP ID Broker mappings (single and multi-AZ) * Add integration targets for CDP Datalakes (single and multi-AZ) * Update testing instructions * Test multi-AZ DL return values * Test single-AZ DL return values * Fix error in cascade value --------- Signed-off-by: Webster Mudge <wmudge@cloudera.com> Signed-off-by: Jim Enright <jenright@cloudera.com> Co-authored-by: Jim Enright <jenright@cloudera.com>
1 parent 2b3029c commit c4f8d30

File tree

54 files changed

+1420
-21
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+1420
-21
lines changed

‎.gitignore‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,12 @@
1919
tests/output
2020

2121
# Remove any integration configuration
22-
tests/integration/integration_config.yml
22+
tests/integration/**/integration_config.yml
23+
tests/integration/inventory
24+
25+
# Ignore Terraform builds
26+
tests/integration/**/.terraform*
27+
tests/integration/**/terraform.tfstate*
2328

2429
# Remove the Sphinx build directory
2530
site/_build

‎tests/TESTING.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Testing
2+
3+
The [ansible-test](https://docs.ansible.com/ansible/latest/dev_guide/developing_collections_testing.html#testing-collections) tool can run a number of testing strategies, notably `sanity`, `units`, and `integration` tests.
4+
5+
## Set up ansible-test
6+
7+
To prepare `ansible-test`, you will need to tackle the following steps:
8+
9+
- Install the collection (and its dependencies) and set the `ANSIBLE_COLLECTIONS_PATHS` accordingly.
10+
- Create a virtual environment
11+
12+
`ansible-test` currently requires Python<=3.10.
13+
14+
You can "go basic":
15+
```bash
16+
python3.10 -m venv ~/.venv/cloud-test
17+
```
18+
19+
Or use your favorite wrapper
20+
```bash
21+
mkvirtualenv -p py310 cloud-test
22+
```
23+
- Install Ansible and testing requirements.
24+
25+
Typically, this should be the latest stable Ansible.
26+
```bash
27+
pip install 'ansible-core>=2.14,<2.15' mock pytest pytest-xdist pytest-mock pytest-forked
28+
```
29+
If you need to test with <2.14, you will need to adjust the `pip` libraries accordingly.
30+
```bash
31+
pip install 'ansible-core>=2.12,<2.13' mock pytest 'pytest-xdist==2.5.0' pytest-mock pytest-forked
32+
```
33+
- Install the collection's Python requirements.
34+
```bash
35+
pip install -r requirements.txt
36+
```
37+
NOTE: if you are also testing changes with `cdpy`, you will want to install that package locally, i.e. `pip install -e <directory to cdpy>`.
38+
39+
## Run the unit tests
40+
41+
Unit tests are implemented in `pytest`.
42+
43+
```bash
44+
ansible-test units --python 3.10
45+
```
46+
47+
## Run the integration tests
48+
49+
The collection integration tests require preexisting infrastructure, e.g. IAM roles.
50+
51+
See the `integration_config.yml.template` for the infrastructure details available and used by the tests. Your infrastructure setup process should produce an `integration_config.yml` file according to these values.
52+
53+
- To see all of the integration tests aka targets:
54+
```bash
55+
ansible-test integration --list-targets
56+
```
57+
- To run a single integration test:
58+
```bash
59+
ansible-test integartion --local datalake_single_az
60+
```
61+
- To run all of the integration tests:
62+
```bash
63+
ansible-test integration --local
64+
```
65+
66+
Once done with the integration tests, you will need to clean up any remaining CDP assets before tearing down the infrastructure (if required by your testing setup).
67+
```bash
68+
ansible-test integration --local --allow-disabled teardown
69+
```

‎tests/integration/integration_config.yml.template‎

Lines changed: 36 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22

3-
# Copyright 2022 Cloudera, Inc. All Rights Reserved.
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
44
#
55
# Licensed under the Apache License, Version 2.0 (the "License");
66
# you may not use this file except in compliance with the License.
@@ -25,11 +25,38 @@
2525
# here will proprogate to all targets, so depending on your testing and development,
2626
# you might want to limit execution to specified targets.
2727

28-
# Specify any environment variables for modules and libraries like CDPCLI, etc.
29-
env_vars:
30-
#CDP_PROFILE: my-cdp-profile
31-
CDP_ACCESS_KEY_ID: "foobarbaz"
32-
CDP_PRIVATE_KEY: "1234567890"
33-
34-
# Parameters used by all functions of cloudera.cloud
35-
env_name: example_env
28+
cdp:
29+
admin_group: "str"
30+
analytics_enabled: "bool"
31+
datalake: "str"
32+
endpoint_access: "str"
33+
environment: "str"
34+
freeipa: "int"
35+
raz_enabled: "bool"
36+
tunnel_enabled: "bool"
37+
user_group: "str"
38+
xaccount_credential: "str"
39+
identity:
40+
datalake_admin_role: "str"
41+
idbroker_role: "str"
42+
log_role: "str"
43+
ranger_audit_role: "str"
44+
xaccount_role: "str"
45+
network:
46+
private_subnets:
47+
- "str"
48+
public_subnets:
49+
- "str"
50+
region: "str"
51+
type: "str"
52+
vpc: "str"
53+
security:
54+
default_group: "str"
55+
knox_group: "str"
56+
ssh:
57+
fingerprint: "str"
58+
name: "str"
59+
public_key: "str"
60+
storage:
61+
datalake_location: "str"
62+
log_location: "str"

‎tests/integration/targets/common/aliases‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
skip/python2
16+
hidden

‎tests/integration/targets/common/defaults/main.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
environment_multi_az: "{{ cdp.environment }}-m-it"
18+
environment_single_az: "{{ cdp.environment }}-s-it"
19+
datalake_multi_az: "{{ cdp.datalake }}-m-it"
20+
datalake_single_az: "{{ cdp.datalake }}-m-it"

‎tests/integration/targets/cred/aliases‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
skip/python2

‎tests/integration/targets/cred/defaults/main.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
xaccount: "{{ cdp.xaccount_credential }}-it"

‎tests/integration/targets/cred/tasks/main.yml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
- name: Describe Credential
18+
cloudera.cloud.env_cred_info:
19+
name: "{{ xaccount }}"
20+
register: __cred
21+
22+
- name: Test for Credential existence
23+
ansible.builtin.assert:
24+
that: __cred.credentials | length == 0
25+
fail_msg: "Test credential already exists: {{ xaccount }}"
26+
27+
- name: Construct a Credential
28+
cloudera.cloud.env_cred:
29+
name: "{{ xaccount }}"
30+
cloud: "{{ network.type }}"
31+
role: "{{ identity.xaccount_role }}"
32+
description: "Integration test"
33+
register: __cred
34+
35+
- name: Test for Credential creation
36+
ansible.builtin.assert:
37+
that:
38+
- __cred.credential | length > 0
39+
- __cred.credential.cloudPlatform == (network.type | upper)
40+
- __cred.credential.credentialName == xaccount
41+
42+
- name: Test idempotent creation of a Credential
43+
cloudera.cloud.env_cred:
44+
name: "{{ xaccount }}"
45+
cloud: "{{ network.type }}"
46+
role: "{{ identity.xaccount_role }}"
47+
description: "Integration test"
48+
register: __cred_idempotent
49+
50+
- name: Test for Credential idempotent state
51+
ansible.builtin.assert:
52+
that:
53+
- __cred_idempotent.credential | length > 0
54+
- __cred.credential.crn == __cred_idempotent.credential.crn
55+
- not __cred_idempotent.changed
56+
57+
- name: Destroy Credential
58+
cloudera.cloud.env_cred:
59+
name: "{{ xaccount }}"
60+
state: absent
61+
register: __cred
62+
63+
- name: Describe Credential after destruction
64+
cloudera.cloud.env_cred_info:
65+
name: "{{ xaccount }}"
66+
register: __cred
67+
68+
- name: Test for Credential existence
69+
ansible.builtin.assert:
70+
that: __cred.credentials | length == 0

‎tests/integration/targets/datalake_multi_az/aliases‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
skip/python2
16+
setup/once/setup_idbroker_multi_az

‎tests/integration/targets/datalake_multi_az/defaults/main.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
3+
# Copyright 2023 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
datalake_multi_az: "{{ cdp.datalake }}-multiaz"

0 commit comments

Comments
 (0)