Skip to content

Commit 1509e20

Browse files
wmudgewmudge
committed
Add initial Molecule framework for Runtime, incl. CDW
Signed-off-by: Webster Mudge <wmudge@cloudera.com>
1 parent fd614ff commit 1509e20

File tree

18 files changed

+796
-0
lines changed

18 files changed

+796
-0
lines changed

‎roles/runtime/molecule/README.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Molecule Testing
2+
3+
Each Molecule scenario represents a networking scheme, e.g. Level0. The default
4+
scheme is Level1, the public/private configuration.
5+
6+
## Dependencies
7+
8+
```bash
9+
pip install "molecule[lint]==3.4"
10+
```
11+
12+
Molecule is configured to execute against `localhost` rather than against a
13+
provisioned `platform`. You will need to ensure that all Python and OS
14+
dependencies are installed, except for the Ansible collections and roles, which
15+
will be gathered during the `dependency` stage.
16+
17+
In short, set up a `cldr-runner` runtime environment without the Ansible
18+
collections and roles.
19+
20+
NOTE: Molecule `3.5` has a bug that prevents the use of collections, so be sure
21+
to pin to `3.4` until this bug is fixed upstream.
22+
23+
## Execution
24+
25+
From within the `platform` role, execute:
26+
27+
* `molecule prepare` to set up the Terraform cloud provider assets
28+
* `molecule converge` to run the `cloudera.exe.platform` role against these
29+
assets
30+
* `molecule cleanup` to tear down the `platform` role and Terraform assets
31+
* or `molecule test` to run the full lifecycle
32+
33+
You can also use `reset` to remove all of the dependencies, e.g. collections,
34+
from the Molecule cache, in order to start fresh.
35+
36+
NOTE: To run other scenarios, i.e. Level0 networking, use the `-s` flag:
37+
`module test -s level0`.
38+
39+
## Configuration
40+
41+
The `molecule.yml` configuration file can accept the following environment
42+
variables:
43+
44+
- `FOUNDRY_NAME_PREFIX`, the "primary key" for the CDP deployment. Defaults to
45+
scenario-specific values.
46+
- `FOUNDRY_INFRA_TYPE`, targeted cloud provider. Defaults to `aws`.
47+
- `FOUNDRY_AWS_REGION`, the AWS region for deployment. Defaults to `us-east-2`.
48+
49+
## SSH Access
50+
51+
The Molecule scenarios create a temporary SSH key for each `prepare`-d run. The
52+
private key is saved to the ephemeral `deployment` directory within the scenerio
53+
parent directory. The SSH key is deleted from the cloud provider during
54+
`cleanup`.
55+
56+
## Terraform
57+
58+
The Molecule scenarios each create a Terraform state directory -- the
59+
`deployment` directory within the scenario parent directory.

‎roles/runtime/molecule/default/collections.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
roles: []
3+
4+
collections:
5+
- name: https://github.com/cloudera-labs/cloudera.cloud
6+
type: git
7+
version: devel
8+
- name: https://github.com/cloudera-labs/cloudera.cluster
9+
type: git
10+
version: main
11+
- amazon.aws
12+
- ansible.netcommon
13+
- ansible.posix
14+
- azure.azcollection
15+
- community.aws
16+
- community.crypto
17+
- community.general
18+
- community.kubernetes
19+
- community.mysql
20+
- community.postgresql
21+
- community.docker
22+
- freeipa.ansible_freeipa
23+
- google.cloud
24+
- kubernetes.core
25+
- netapp.azure

‎roles/runtime/molecule/default/molecule.yml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
dependency:
3+
name: galaxy
4+
enabled: true
5+
options:
6+
force: false
7+
driver:
8+
name: delegated
9+
options:
10+
managed: false
11+
ansible_connection_options:
12+
ansible_connection: local
13+
platforms:
14+
- name: placeholder
15+
provisioner:
16+
name: ansible
17+
log: true
18+
config_options:
19+
defaults:
20+
callback_whitelist: profile_tasks, timer, yaml
21+
enable_task_debugger: true
22+
ssh_connection:
23+
pipelining: false
24+
vvv: true
25+
playbooks:
26+
prepare: ../shared/prepare.yml
27+
converge: ../shared/converge.yml
28+
cleanup: ../shared/cleanup.yml
29+
inventory:
30+
host_vars:
31+
localhost:
32+
globals:
33+
admin_password: "@NotSecure456!"
34+
name_prefix: ${FOUNDRY_NAME_PREFIX:-r01}
35+
infra_type: ${FOUNDRY_INFRA_TYPE:-aws}
36+
infra:
37+
vpc:
38+
name: ${FOUNDRY_NAME_PREFIX:-r01}-test
39+
storage:
40+
name: ${FOUNDRY_NAME_PREFIX:-r01}-test
41+
security_group:
42+
knox:
43+
name: ${FOUNDRY_NAME_PREFIX:-r01}-test-knox
44+
default:
45+
name: ${FOUNDRY_NAME_PREFIX:-r01}-test-default
46+
aws:
47+
region: ${FOUNDRY_AWS_REGION:-us-east-2}
48+
env:
49+
tunnel: yes # L1 Networking
50+
public_endpoint_access: yes
51+
dw:
52+
force_delete: yes # Non-force delete is inconsistent until we can filter 'compactor' VWs
53+
tags:
54+
project: "${FOUNDRY_NAME_PREFIX:-r01}-CDW-testing"
55+
definitions:
56+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-empty
57+
- virtual_warehouses:
58+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-dbc-default
59+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-dbc-hive
60+
type: hive
61+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-dbc-impala
62+
type: impala
63+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-dbc-autoscale
64+
autoscaling:
65+
min_nodes: 0
66+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-named
67+
virtual_warehouses:
68+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-named-default
69+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-named-hive
70+
type: hive
71+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-named-impala
72+
type: impala
73+
- name: ${FOUNDRY_NAME_PREFIX:-r01}-named-autoscale
74+
autoscaling:
75+
min_nodes: 0
76+
verifier:
77+
name: ansible
78+
lint: |
79+
set -e
80+
yamllint .
81+
ansible-lint .

‎roles/runtime/molecule/default/requirements.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
---
2+
roles: []

‎roles/runtime/molecule/default/terraform/level1.tf‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
terraform {
2+
required_version = ">= 0.13"
3+
required_providers {
4+
aws = "~> 3.0"
5+
}
6+
}
7+
8+
provider "aws" {
9+
region = var.region
10+
default_tags {
11+
tags = var.tags
12+
}
13+
ignore_tags {
14+
keys = ["kubernetes.io/role/internal-elb", "kubernetes.io/role/elb"]
15+
key_prefixes = ["kubernetes.io/cluster/"]
16+
}
17+
}
18+
19+
data "aws_availability_zones" "available" {
20+
state = "available"
21+
}
22+
23+
module "vpc" {
24+
source = "terraform-aws-modules/vpc/aws"
25+
version = "3.2.0"
26+
27+
tags = var.tags
28+
29+
name = var.vpc_name
30+
cidr = "10.10.0.0/16"
31+
32+
enable_nat_gateway = true
33+
igw_tags = var.tags
34+
nat_eip_tags = var.tags
35+
nat_gateway_tags = var.tags
36+
37+
enable_dns_hostnames = true
38+
39+
# azs = [ "us-east-2a", "us-east-2b", "us-east-2c"]
40+
azs = data.aws_availability_zones.available.names
41+
public_subnets = [ "10.10.0.0/19", "10.10.32.0/19", "10.10.64.0/19" ]
42+
public_subnet_tags = merge({ "kubernetes.io/role/elb" = "1" }, var.tags)
43+
public_route_table_tags = var.tags
44+
private_subnets = [ "10.10.96.0/19", "10.10.128.0/19", "10.10.160.0/19" ]
45+
private_subnet_tags = merge({ "kubernetes.io/role/internal-elb" = "1" }, var.tags)
46+
private_route_table_tags = var.tags
47+
}
48+
49+
module "default_sg" {
50+
source = "terraform-aws-modules/security-group/aws"
51+
52+
name = var.sg_names.default
53+
use_name_prefix = false
54+
description = "Default for Molecule integration test. Namespace: ${var.name_prefix}"
55+
vpc_id = module.vpc.vpc_id
56+
57+
ingress_cidr_blocks = [ module.vpc.vpc_cidr_block ]
58+
ingress_rules = [ "all-all" ]
59+
ingress_with_cidr_blocks = var.extra_rules
60+
ingress_with_self = [ { rule = "all-all" } ]
61+
62+
egress_cidr_blocks = [ "0.0.0.0/0" ]
63+
egress_rules = [ "all-all" ]
64+
}
65+
66+
module "knox_sg" {
67+
source = "terraform-aws-modules/security-group/aws"
68+
69+
name = var.sg_names.knox
70+
use_name_prefix = false
71+
description = "Knox for Molecule integration test. Namespace: ${var.name_prefix}"
72+
vpc_id = module.vpc.vpc_id
73+
74+
ingress_cidr_blocks = [ module.vpc.vpc_cidr_block ]
75+
ingress_rules = [ "all-all" ]
76+
ingress_with_cidr_blocks = var.extra_rules
77+
ingress_with_self = [ { rule = "all-all" } ]
78+
79+
egress_cidr_blocks = [ "0.0.0.0/0" ]
80+
egress_rules = [ "all-all" ]
81+
}

‎roles/runtime/molecule/default/terraform/storage.tf‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
module "s3_bucket" {
2+
source = "terraform-aws-modules/s3-bucket/aws"
3+
4+
bucket = var.s3_bucket_name
5+
acl = "private"
6+
force_destroy = true
7+
}
8+
9+
resource "aws_s3_bucket_object" "cml" {
10+
bucket = module.s3_bucket.s3_bucket_id
11+
key = "datasci/"
12+
content_type = "application/x-directory"
13+
}
14+
15+
resource "aws_s3_bucket_object" "cde" {
16+
bucket = module.s3_bucket.s3_bucket_id
17+
key = "dataeng/"
18+
content_type = "application/x-directory"
19+
}
20+
21+
resource "aws_s3_bucket_object" "log" {
22+
bucket = module.s3_bucket.s3_bucket_id
23+
key = "logs/"
24+
content_type = "application/x-directory"
25+
}
26+
27+
resource "aws_s3_bucket_object" "data" {
28+
bucket = module.s3_bucket.s3_bucket_id
29+
key = "data/"
30+
content_type = "application/x-directory"
31+
}
32+
33+
resource "aws_s3_bucket_object" "audit" {
34+
bucket = module.s3_bucket.s3_bucket_id
35+
key = "ranger/audit/"
36+
content_type = "application/x-directory"
37+
}

‎roles/runtime/molecule/default/terraform/vars.tf‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
variable "name_prefix" {
2+
type = string
3+
description = "An unique identifer"
4+
}
5+
6+
variable "region" {
7+
type = string
8+
description = "AWS region name"
9+
}
10+
11+
variable "vpc_name" {
12+
type = string
13+
description = "AWS VPC name"
14+
}
15+
16+
variable "sg_names" {
17+
type = object({
18+
knox = string
19+
default = string
20+
})
21+
description = "AWS Security Group names for Knox and Default"
22+
}
23+
24+
variable "tags" {
25+
type = map(string)
26+
description = "Key-Value pairs of tags applied to AWS assets"
27+
}
28+
29+
variable "extra_rules" {
30+
type = list(map(any))
31+
description = "A list of maps representing the additional rules added to the Knox and Default security groups"
32+
default = []
33+
}
34+
35+
variable "s3_bucket_name" {
36+
type = string
37+
description = "AWS S3 bucket name for logs, audit, and data (all-in-one)"
38+
}

‎roles/runtime/molecule/default/verify.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
# This is an example playbook to execute Ansible tests.
3+
4+
- name: Verify
5+
hosts: all
6+
gather_facts: false
7+
tasks:
8+
- name: Example assertion
9+
assert:
10+
that: true

‎roles/runtime/molecule/level0/collections.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
---
2+
3+
collections:
4+
- name: https://github.com/cloudera-labs/cloudera.cloud
5+
type: git
6+
version: devel
7+
- name: https://github.com/cloudera-labs/cloudera.cluster
8+
type: git
9+
version: main
10+
- amazon.aws
11+
- ansible.netcommon
12+
- ansible.posix
13+
- azure.azcollection
14+
- community.aws
15+
- community.crypto
16+
- community.general
17+
- community.kubernetes
18+
- community.mysql
19+
- community.postgresql
20+
- community.docker
21+
- freeipa.ansible_freeipa
22+
- google.cloud
23+
- kubernetes.core
24+
- netapp.azure

0 commit comments

Comments
 (0)