Skip to content

Commit 60304ad

Browse files
wmudgeChaffelson
wmudge
and
Chaffelson
authored
Update with collected CDP PVC changes (#107)
* Update with Private Cloud prerequisite and Control Plane changes (#61) * Add control keys for autotls, pvc_type, free_ipa to control deployment behavior more simply. * Standardise the host group name for ECS nodes to be 'ecs_nodes' to match the other standard groups we use in Ansible inventory * Deprecate duplicate filter_null_configs filter from api_client * Add handler to restart cloudera management service * Migrate autotls implementation * Migrate cms_tls setup * Add default external_auth configuration to generally handle freeipa or mit setup * Update importAdminCredentials command to not fail when already imported, but report other errors * Add new role cloudera_manager.services_info to perform useful service discovery on existing clusters. * Migrate role to set session_timeout for cloudera_manager * Migrate role to set hue_ticket_lifetime for PvC-DS deployments * Migrate role to setup TLS for KMS * Migrate role to fix some libs for the Oozie UI in some PvC-DS deployments * Migrate role to setup some default Ranger policies for some PvC-DS deployments * Migrate role to setup a SOLR role in Knox for some PvC-DS deployments * Migrate role to ensure a Ranger plugin for SOLR is deployed in some PvC-DS deployments * Update the defaults for database type and version to respond to el7 or el8 appropriately. * Migrate role to setup WXM. * Update krb5_client deployment for FreeIPA setup, including a patch for dbus_session config and specific configs for when running PvC-DS. * Default krb5_domain to krb5_realm.lower automatically. * Add default kerberos configuration to krb5_common, including simple defaults for when MIT KDC or Red Hat IPA are selected. Passwords default to the cloudera_manager_admin_password instead of hardcoded values like 'changeme' * Add fixes for FreeIPA server deployment * Fix refresh_ranger_kms_repo role to function correctly when determining the Ranger URL in modern Ansible. * Add operation to restart a given cluster or a given cluster's services or cluster management services for user convenience. They could be handlers, but this felt more useful as more people know how to use roles than handlers. * Migrate role to setup iptables or nftables for PvC ECS deployment on Rhel7 or Rhel8 * Add firewalld to unwanted services during automated os prereq setup * Add fix where setting up postgresql_connector sometimes requires python3-psycopg2 to be setup for SSB. * Migrate role to set up a subset of necessary local accounts on ecs_nodes * Add check to ensure that FreeIPA and a custom repo are not on the same host as they both try to hardcode port 8443. * Enhance error message when TLS setup is only being partially applied to hosts in the cluster definition * Update ecs cluster template to set version to DATA_SERVICES1 to reflect current Cloudera Manager 7.6.5 requirements * Modifiy ecs services Jinja template to seek host groups by long name. * Explicitly set default database_type to postgresql to avoid user confusion * Add nfs-utils to OS prereqs when installing ECS * Add control for whether or not embedded database mode for ECS is implemented * Remove unused deployment.j2 template * Add controlPlaneValuesEmbedded.j2 for embedded database values * Fix bug in services.j2 for ECS deployment where it would look for the wrong host template name * Rename free_ipa switch to freeipa_activated to match other top level switches * Allow Cloudera Manager version and distro to be set explicitly for repo setup * Update default cloudera-manager version to 7.6.5 * Fix custom_repo to recognise ecs_nodes as valid * Update dbus patch for freeipa client to only restart services if something is changed * Add autodns support to freeipa clients * Add autodns function to freeipa server setup, including creating required zones and records for PvC-DS ECS if that is being deployed * Add task to Flush and Delete IPTables when setting up ECS * Set default Cloudera Manager version to 7.6.1 for base deployments. (7.6.5 is primarily for PvC-DS.) * Add draft ECS teardown processes * Add cloudera.cluster.operations.stop_cluster as a convenience method, as ECS needs to be stopped and cleaned in a specific sequence. * Provide additional wildcard DNS records for ECS in FreeIPA Autodns setup * Fixes for RHEL8.6 support and custom_repo with Cloudera Manager (#83) * Fix download and reuse of Cloudera Manager repo-as-tarball * Switch to using custom_repo as base url for archive.cloudera.com by default if custom_repo is specified in build without a specific base_url being supplied. * Fix kts setup for RHEL8+ where gpg 2.1+ is used which has changed the default file set - look for kbx files. * Setup of automatic DNS on the freeipa server now supports running on ec2 instances with RHEL8.6, and EL8 generally * Automatically set selinux to permissive on the krb5_server for RHEL8, as otherwise setup is blocked * Separate Py2 and Py3 setup on RHEL8 in preparation for final Py2 deprecation in Cloudera products * Set default admins group as configurable variable in freeipa config * Add option to ensure creation of a 'superuser' in FreeIPA as the default 'admin' user can clash with system users within CDP in some circumstances. This user is then useful to PvC-DS installs and not created by default otherwise. * Create Cloudera Manager module framework (#62) * Create common cm_utils.py for managing CM API interactions, unit and integration test frameworks, and cm_version_info and cm_endpoint_info modules. * Clean up error messages and remove unused imports * Create ad-hoc/unimplemented API resource module, cm_resource_info * Fix JSON decode bug for ApiExceptions * Centralize call_api method * Add documentation and document fragment for cm_resource * Create cm_resource module for ad-hoc CM API endpoint calls * Update to handle private IP installations (#93) * Handle delegation to CM server when needed * Handle remote CA management for sidecar/embedded FreeIPA installs Signed-off-by: Daniel Chaffelson <chaffelson@gmail.com> Signed-off-by: Webster Mudge <wmudge@cloudera.com> Co-authored-by: Dan Chaffelson <chaffelson@gmail.com>
1 parent 32fa36c commit 60304ad

File tree

136 files changed

+4352
-156
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

136 files changed

+4352
-156
lines changed

‎.gitignore‎

Lines changed: 164 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,164 @@
1-
.idea
2-
.vscode
3-
test*
4-
*.pyc
5-
*.bak
6-
.DS_Store
7-
venv
1+
# Copyright 2022 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Keep the Galaxy builds
16+
!cloudera-cluster-*.tar.gz
17+
18+
# Ignore the test output
19+
tests/output
20+
21+
# Remove any integration configuration
22+
tests/integration/integration_config.yml
23+
24+
# Remove the Sphinx build directory
25+
site/_build
26+
27+
### Python template
28+
# Byte-compiled / optimized / DLL files
29+
__pycache__/
30+
*.py[cod]
31+
*$py.class
32+
33+
# C extensions
34+
*.so
35+
36+
# Distribution / packaging
37+
.Python
38+
build/
39+
develop-eggs/
40+
dist/
41+
downloads/
42+
eggs/
43+
.eggs/
44+
lib/
45+
lib64/
46+
parts/
47+
sdist/
48+
var/
49+
wheels/
50+
pip-wheel-metadata/
51+
share/python-wheels/
52+
*.egg-info/
53+
.installed.cfg
54+
*.egg
55+
MANIFEST
56+
57+
# PyInstaller
58+
# Usually these files are written by a python script from a template
59+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
60+
*.manifest
61+
*.spec
62+
63+
# Installer logs
64+
pip-log.txt
65+
pip-delete-this-directory.txt
66+
67+
# Unit test / coverage reports
68+
htmlcov/
69+
.tox/
70+
.nox/
71+
.coverage
72+
.coverage.*
73+
.cache
74+
nosetests.xml
75+
coverage.xml
76+
*.cover
77+
*.py,cover
78+
.hypothesis/
79+
.pytest_cache/
80+
cover/
81+
82+
# Translations
83+
*.mo
84+
*.pot
85+
86+
# Django stuff:
87+
*.log
88+
local_settings.py
89+
db.sqlite3
90+
db.sqlite3-journal
91+
92+
# Flask stuff:
93+
instance/
94+
.webassets-cache
95+
96+
# Scrapy stuff:
97+
.scrapy
98+
99+
# Sphinx documentation
100+
docs/_build/
101+
102+
# PyBuilder
103+
.pybuilder/
104+
target/
105+
106+
# Jupyter Notebook
107+
.ipynb_checkpoints
108+
109+
# IPython
110+
profile_default/
111+
ipython_config.py
112+
113+
# pyenv
114+
# For a library or package, you might want to ignore these files since the code is
115+
# intended to run in multiple environments; otherwise, check them in:
116+
# .python-version
117+
118+
# pipenv
119+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
120+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
121+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
122+
# install all needed dependencies.
123+
#Pipfile.lock
124+
125+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
126+
__pypackages__/
127+
128+
# Celery stuff
129+
celerybeat-schedule
130+
celerybeat.pid
131+
132+
# SageMath parsed files
133+
*.sage.py
134+
135+
# Environments
136+
.env
137+
.venv
138+
env/
139+
venv/
140+
ENV/
141+
env.bak/
142+
venv.bak/
143+
144+
# Spyder project settings
145+
.spyderproject
146+
.spyproject
147+
148+
# Rope project settings
149+
.ropeproject
150+
151+
# mypy
152+
.mypy_cache/
153+
.dmypy.json
154+
dmypy.json
155+
156+
# Pyre type checker
157+
.pyre/
158+
159+
# pytype static type analyzer
160+
.pytype/
161+
162+
# Cython debug symbols
163+
cython_debug/
164+

‎meta/runtime.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
3+
# Copyright 2022 Cloudera, Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
requires_ansible: ">=2.10"

‎plugins/doc_fragments/cm_endpoint.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*-
3+
4+
# Copyright 2022 Cloudera, Inc. All Rights Reserved.
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
class ModuleDocFragment(object):
19+
DOCUMENTATION = r'''
20+
options:
21+
url:
22+
description:
23+
- The CM API endpoint URL and should include scheme, host, port, and API root path.
24+
- Mutually exclusive with I(host).
25+
type: str
26+
required: False
27+
aliases:
28+
- endpoint
29+
- cm_endpoint_url
30+
'''

‎plugins/doc_fragments/cm_options.py‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*-
3+
4+
# Copyright 2022 Cloudera, Inc. All Rights Reserved.
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
class ModuleDocFragment(object):
19+
DOCUMENTATION = r'''
20+
options:
21+
host:
22+
description:
23+
- Hostname of the CM API endpoint.
24+
- If set, the C(host) parameter will trigger CM API endpoint discovery, which will follow redirects.
25+
- Mutually exclusive with I(url).
26+
type: str
27+
required: False
28+
aliases:
29+
- hostname
30+
port:
31+
description:
32+
- Port of the CM API endpoint.
33+
- If set, CM API endpoint discovery will connect to the designated port first and will follow redirects.
34+
type: int
35+
required: False
36+
default: 7180
37+
version:
38+
description:
39+
- API version of the CM API endpoint.
40+
type: str
41+
required: False
42+
default: True
43+
aliases:
44+
- tls
45+
force_tls:
46+
description:
47+
- Flag to force TLS during CM API endpoint discovery.
48+
- If C(False), discovery will first try HTTP and follow any redirects.
49+
type: bool
50+
required: False
51+
default: False
52+
verify_tls:
53+
description:
54+
- Verify the TLS certificates for the CM API endpoint.
55+
type: bool
56+
required: False
57+
default: True
58+
aliases:
59+
- tls
60+
username:
61+
description:
62+
- Username for access to the CM API endpoint.
63+
type: str
64+
required: True
65+
password:
66+
description:
67+
- Password for access to the CM API endpoint.
68+
- This parameter is set to C(no_log).
69+
type: str
70+
required: True
71+
debug:
72+
description:
73+
- Capture the HTTP interaction logs with the CM API endpoint.
74+
type: bool
75+
required: False
76+
default: False
77+
aliases:
78+
- debug_endpoints
79+
agent_header:
80+
description:
81+
- Set the HTTP user agent header when interacting with the CM API endpoint.
82+
type: str
83+
required: False
84+
default: ClouderaFoundry
85+
'''

‎plugins/doc_fragments/cm_resource.py‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*-
3+
4+
# Copyright 2022 Cloudera, Inc. All Rights Reserved.
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
class ModuleDocFragment(object):
19+
DOCUMENTATION = r'''
20+
options:
21+
path:
22+
description:
23+
- Path of the CM API endpoint call.
24+
type: str
25+
required: True
26+
query:
27+
description:
28+
- HTTP query parameters for the CM API endpoint call.
29+
type: dict
30+
aliases:
31+
- query_parameters
32+
- parameters
33+
field:
34+
description:
35+
- Field within the response for result extraction.
36+
- Use I(field) when the returned object has an enclosing field.
37+
type: str
38+
default: 'items'
39+
aliases:
40+
- return_field
41+
'''

0 commit comments

Comments
 (0)