Skip to content

Commit 7e2c7e6

Browse files
wmudgewmudge
authored
Update README for 3.0.0 (#281)
Signed-off-by: Webster Mudge <wmudge@cloudera.com>
1 parent fa42acc commit 7e2c7e6

File tree

1 file changed

+43
-167
lines changed

1 file changed

+43
-167
lines changed

‎README.md‎

Lines changed: 43 additions & 167 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,34 @@
1-
# cloudera.exe - Runlevel Management and Utilities for Cloudera Data Platform (CDP)
1+
# cloudera.exe - Tools and Utilities for Cloudera Deployments
22

33
[![API documentation](https://github.com/cloudera-labs/cloudera.exe/actions/workflows/publish_docs.yml/badge.svg?branch=main&event=push)](https://github.com/cloudera-labs/cloudera.exe/actions/workflows/publish_docs.yml)
44

5-
`cloudera.exe` is an Ansible collection that offers runlevel management of your **[Cloudera Data Platform (CDP)](https://www.cloudera.com/products/cloudera-data-platform.html) Public Cloud and Private Cloud** deployments. The collection contains a number of utilities for common scenarios encountered when managing a CDP deployment, including:
6-
* Set up and management of external dependencies, e.g. database, Kerberos, LDAP
7-
* Execution of Common deployment sequences (via importable playbooks)
5+
`cloudera.exe` is an Ansible collection for helping setup **[Cloudera Data Platform (CDP)](https://www.cloudera.com/products/cloudera-data-platform.html) on cloud (Public Cloud) and on premise (Private Cloud)** deployments. The collection contains a number of utilities for common scenarios encountered when managing a deployment, including:
86

9-
The collection is unabashedly an _opinionated_ approach of managing your CDP resources - it's resources can be used to set up your CDP infrastructure, configure the host machines, install and configure CDP and its services, and more. The collection interacts across several control planes from CDP Public Cloud and cloud provider endpoints to Cloudera Manager for CDP Private Cloud and Public Cloud Data Hubs. In short, it has opinions about how to get things done. If you are looking for automation resources that _only_ interact with CDP resources - that is, assets that are focused solely on Cloudera software - please look at [`cloudera.cloud` for Public Cloud](https://github.com/cloudera-labs/cloudera.cloud) and [`cloudera.cluster` for Private Cloud and Cloudera Manager](https://github.com/cloudera-labs/cloudera.cluster).
7+
* Supporting databases
8+
* Kerberos
9+
* LDAP
10+
* TLS
11+
* Cloudera Manager Server and Agent binaries
12+
* Repositories
13+
* OS kernel best practices
1014

11-
Core to the collection is the [configuration file](./docs/configuration.yml) which many of the collection's roles use as a central "switchboard" for their functions. The collection works hand-in-hand with the [`cloudera-deploy` application](https://github.com/cloudera-labs/cloudera-deploy/) to execute _definitions_ which include variations on this configuration; many of the functions in `cloudera-deploy` have relocated to this collection to streamline its use.
15+
The collection is unabashedly an _opinionated_ approach of managing your resources - its resources can be used to configure the host machines, install and configure Cloudera and its services, and more. To be clear, **these resources are OPTIONAL**, yet they are helpful for greenfield, developer-centric, and quickstart deployments. The collection interacts with cloud provider endpoints to host systems. In short, it has opinions about how to get things done.
1216

13-
The collection provides _playbooks_, _roles_, and _plugins_ for working with CDP deployments. Notably, the playbooks encapsulate typical set up and tear down deployment operations, aka runlevels:
14-
15-
| Name | Description |
16-
| --- | --- |
17-
| `pbc_infra_setup.yml` | Public Cloud infrastructure setup (AWS, Azure, GCP), using either Terraform or Ansible |
18-
| `pbc_infra_teardown.yml` | Public Cloud infrastructure teardown (AWS, Azure, GCP), using either Terraform or Ansible |
19-
| `pbc_setup.yml` | Public Cloud Datalake and Data Services setup |
20-
| `pbc_teardown.yml` | Public Cloud Datalake and Data Services teardown |
21-
| `pvc_base_postfix.yml` | Private Cloud setup, postfix |
22-
| `pvc_base_prereqs_ext.yml` | Private Cloud external dependencies, e.g. JVM, Kerberos, database |
23-
| `pvc_base_prereqs_int.yml` | Private Cloud internal dependencies, e.g. Cloudera Manager server and agent install |
24-
| `pvc_base_setup.yml` | Private Cloud cluster setup |
25-
| `pvc_base_teardown.yml` | Private Cloud cluster teardown |
26-
27-
`cloudera.exe`-powered applications, like `cloudera-deploy`, import these playbooks to enable these runlevel operations.
28-
29-
The other collection assets - the _roles_ and _plugins_ - are detailed in the [API documentation](https://cloudera-labs.github.io/cloudera.exe/). While these resource can be used separately, most expect the common configuration noted above and a sequence of execution defined within the noted playbooks.
17+
If you are looking for automation resources that _only_ interact with Cloudera resources - that is, assets that are focused solely on Cloudera software - please look at [`cloudera.cloud`](https://github.com/cloudera-labs/cloudera.cloud) and [`cloudera.cluster`](https://github.com/cloudera-labs/cloudera.cluster).
3018

3119
## Quickstart
3220

21+
See the [API documentation](https://cloudera-labs.github.io/cloudera.exe/) for details for each plugin and role within the collection.
22+
3323
1. [Install the collection](#installation)
3424
2. [Install the requirements](#requirements)
3525
3. [Use the collection](#using-the-collection)
3626

37-
## API
38-
39-
See the [API documentation](https://cloudera-labs.github.io/cloudera.exe/) for details for each plugin and role within the collection.
40-
4127
## Roadmap
4228

4329
If you want to see what we are working on or have pending, check out:
4430

45-
* the [Milestones](https://github.com/cloudera-labs/cloudera.exe/milestones) and [active issues](https://github.com/cloudera-labs/cloudera.exe/issues?q=is%3Aissue+is%3Aopen+milestone%3A*) to see our current activity,
31+
* the [Milestones](https://github.com/cloudera-labs/cloudera.exe/milestones) and [active issues](https://github.com/cloudera-labs/cloudera.exe/issues?q=is%3Aissue+is%3Aopen+milestone%3A*) to see our current activity,
4632
* the [issue backlog](https://github.com/cloudera-labs/cloudera.exe/issues?q=is%3Aopen+is%3Aissue+no%3Amilestone) to see what work is pending or under consideration, and
4733
* read up on the [Ideas](https://github.com/cloudera-labs/cloudera.exe/discussions/categories/ideas) we have in mind.
4834

@@ -54,11 +40,16 @@ For more information on how to get involved with the `cloudera.exe` Ansible coll
5440

5541
## Installation
5642

57-
To install the `cloudera.exe` collection, you have several options. Please note that to date, we have not yet published this collection to the public Ansible Galaxy server, so you cannot install it via direct namespace declaration, rather you must specify a Git project and (optionally) branch.
43+
To install the `cloudera.exe` collection, you have several options.
5844

59-
### Option #1: Install from GitHub
45+
The preferred method is to install via Ansible Galaxy; in your `requirements.yml` file, add the following:
6046

61-
Create or edit the `requirements.yml` file in your project with the
47+
```yaml
48+
collections:
49+
- name: cloudera.exe
50+
```
51+
52+
If you want to install from GitHub, add to your `requirements.yml` file:
6253
following:
6354

6455
```yaml
@@ -77,92 +68,15 @@ ansible-galaxy collection install -r requirements.yml
7768
You can also install the collection directly:
7869

7970
```bash
80-
ansible-galaxy collection install git+https://github.com/cloudera-labs/cloudera.exe.git@main
71+
# From Ansible Galaxy
72+
ansible-galaxy collection install cloudera.exe
8173
```
8274

83-
### Option #2: Install the tarball
84-
85-
Periodically, the collection is packaged into a distribution which you can
86-
install directly:
87-
8875
```bash
89-
ansible-galaxy collection install <collection-tarball>
76+
# From GitHub
77+
ansible-galaxy collection install git+https://github.com/cloudera-labs/cloudera.exe.git@main
9078
```
9179

92-
See [Building the Collection](#building-the-collection) for details on creating a local tarball.
93-
94-
## Requirements
95-
96-
The `cloudera.exe` expects `ansible-core>=2.10,<2.13`.
97-
98-
> [!WARNING]
99-
> The current functionality of the `cloudera.cluster` dependency does not yet work with Ansible version `2.13` and later.
100-
101-
The collection has the following _required_ dependencies:
102-
103-
| Name | Type | Version |
104-
|------|------|---------|
105-
| `cloudera.cloud` | collection | `main` |
106-
| `cloudera.cluster` | collection | `main` |
107-
| `ansible.netcommon` | collection | `2.5.1` |
108-
| `community.general` | collection | `4.5.0` |
109-
110-
You will need to add the following, depending on your target deployment, but all are collectively _optional_ dependencies:
111-
112-
**Private Cloud**
113-
114-
See the [requirements for `cloudera-labs/cloudera.cluster`](https://github.com/wmudge/cloudera.cluster#requirements) for details.
115-
116-
| Name | Type | Version |
117-
|------|------|---------|
118-
| `community.mysql` | collection | `3.8.0` |
119-
| `community.postgresql` | collection | `3.3.0` |
120-
| `freeipa.ansible_freeipa` | collection | `1.11.1` |
121-
| `geerlingguy.postgresql` | role | `3.3.0` |
122-
| `geerlingguy.mysql` (patched) | role | `master` |
123-
124-
**Terraform**
125-
126-
If you intend to use Terraform as your infrastructure engine within the `cloudera.exe.infra` role, then install the following:
127-
128-
| Name | Type | Version |
129-
|------|------|---------|
130-
| `cloud.terraform` | collection | `1.1.1` |
131-
132-
**AWS**
133-
134-
See the [AWS Execution Environment configuration](https://github.com/cloudera-labs/cldr-runner/blob/main/aws/execution-environment.yml) in `cloudera-labs/cldr-runner` for details on setting up the Python and system requirements.
135-
136-
| Name | Type | Version |
137-
|------|------|---------|
138-
| `amazon.aws` | collection | `3.0.0` |
139-
| `community.aws` | collection | `3.0.1` |
140-
141-
**Azure**
142-
143-
See the [Azure Execution Environment configuration](https://github.com/cloudera-labs/cldr-runner/blob/main/azure/execution-environment.yml) in `cloudera-labs/cldr-runner` for details on setting up the Python and system requirements.
144-
145-
| Name | Type | Version |
146-
|------|------|---------|
147-
| `azure.azcollection` | collection | `1.11.0` |
148-
| `netapp.azure` | collection | `21.10.0` |
149-
150-
**GCP**
151-
152-
See the [GCP Execution Environment configuration](https://github.com/cloudera-labs/cldr-runner/blob/main/gcp/execution-environment.yml) in `cloudera-labs/cldr-runner` for details on setting up the Python and system requirements.
153-
154-
| Name | Type | Version |
155-
|------|------|---------|
156-
| `google.cloud` | collection | `1.0.2` |
157-
158-
The collection also requires the following Python libraries to operate its modules and tasks:
159-
160-
* [netaddr](https://pypi.org/project/netaddr/)
161-
162-
The collection's Python dependencies alone, _not_ the required Python libraries of its collection dependencies, are in `requirements.txt`.
163-
164-
All collection dependencies, required and optional, can be found in `requirements.yml`; only the _required_ **non-Cloudera** dependencies are in `galaxy.yml`. `ansible-galaxy` will install only the _required_ **non-Cloudera** collection dependencies; you will need to add `cloudera.cloud`, `cloudera.cluster`, and the _optional_ collection dependencies as needed (see above).
165-
16680
`ansible-builder` can discover and install all Python dependencies - current collection and dependencies - if you wish to use that application to construct your environment. Otherwise, you will need to read each collection and role dependency and follow its installation instructions.
16781

16882
See the [Collection Metadata](https://ansible.readthedocs.io/projects/builder/en/latest/collection_metadata/) section for further details on how to install (and manage) collection dependencies.
@@ -171,76 +85,38 @@ You may wish to use a _virtual environment_ to manage the Python dependencies.
17185

17286
## Using the Collection
17387

174-
This collection is designed to work hand-in-hand with the [`cloudera-deploy` application](https://github.com/cloudera-labs/cloudera-deploy), which uses the reference playbooks in the `playbooks` directory to drive the operations of its example definitions.
88+
This collection is designed to help you get up and running with Cloudera on cloud and on premise. It is decidedly _opinionated_ -- that is, these roles and plugins make assumes as to how certain configurations and requirements are met. **THESE RESOURCES ARE COMPLETELY OPTIONAL. THEY EXIST ONLY TO ASSIST YOU WITH BOOTSTRAPPING AND GREENFIELD EXAMPLES!**
89+
90+
Feel free to use these resources as needed!
17591

176-
Once installed, reference the collection in your playbooks and roles.
92+
Once installed, reference the collection in playbooks and roles.
17793

17894
For example, here we use the
179-
[`cloudera.exe.init_deployment` role](https://github.com/cloudera-labs/cloudera.exe/tree/main/roles/init_deployment) to read the configuration details and then import the Public Cloud playbooks to set up and provision an Environment and Datalake:
95+
[`cloudera.exe.cm_agent` role](https://github.com/cloudera-labs/cloudera.exe/tree/main/roles/cm_agent) to download and install the Cloudera Manager agent software from the Cloudera Archive repository:
18096

18197
```yaml
182-
- name: Marshal the variables
183-
hosts: localhost
184-
connection: local
98+
- name: Install the CM agent
99+
hosts: cluster_hosts
185100
gather_facts: yes
186101
tasks:
187-
- name: Read definition variables
188-
ansible.builtin.include_role:
189-
name: cloudera.exe.init_deployment
190-
public: yes
191-
when: init__completed is undefined
192-
tags:
193-
- always
194-
195-
- name: Set up CDP Public Cloud infrastructure (Ansible-based)
196-
ansible.builtin.import_playbook: cloudera.exe.pbc_infra_setup.yml
197-
198-
- name: Set up CDP Public Cloud (Env and DL example)
199-
ansible.builtin.import_playbook: cloudera.exe.pbc_setup.yml
102+
- name: Install the agent and register with Cloudera Manager
103+
ansible.builtin.import_role:
104+
name: cloudera.exe.cm_agent
105+
vars:
106+
cloudera_manager_host: cm.example.internal
200107
```
201108

202-
> [!IMPORTANT]
203-
> You **must** run `cloudera.exe.init_deployment` before calling any of the collection's playbooks. This call must occur within the source project, otherwise Ansible's `playbook_dir` will change to the collection's installation directory and variable lookups might not work as expected.
204-
205-
### Legacy Execution Modes
206-
207-
> [!WARNING]
208-
> These documents and their modes of operation are deprecated in version 2.x. For example, the use of Ansible tags to trigger coarse runlevels have been replaced by explicit playbook execution. However, the "inner" tag structures still remain and might be relevant to some execution modes.
209-
210-
See the [execution examples](docs/runlevels.md#execution) in the Deployment Runlevels document.
211-
212-
For more information on the collection, check out:
213-
214-
+ [Configuration Guide](docs/configuration.md)
215-
+ [Runlevels Guide](docs/runlevels.md)
216-
+ [Architecture and Design Guide](docs/design.md)
109+
## Building the API Documentation
217110

218-
## Building the Collection
111+
If you wish to create a local copy of the API documentation, first set up the `hatch` build tool, as shown in the [TESTING](./TESTING.md) guide.
219112

220-
To create a local collection tarball, run:
113+
Then, you can run:
221114

222115
```bash
223-
ansible-galaxy collection build
116+
hatch run docs:build
224117
```
225118

226-
## Building the API Documentation
227-
228-
To create a local copy of the API documentation, first make sure the collection is in your `ANSIBLE_COLLECTIONS_PATHS`. Then run the following:
229-
230-
```bash
231-
# change into the /docsbuild directory
232-
cd docsbuild
233-
234-
# install the build requirements (antsibull-docs); you may want to set up a
235-
# dedicated virtual environment
236-
pip install ansible-core https://github.com/cloudera-labs/antsibull-docs/archive/cldr-docsite.tar.gz
237-
238-
# Install the collection's build dependencies
239-
pip install requirements.txt
240-
241-
# Then run the build script
242-
./build.sh
243-
```
119+
This will kick off the build toolchain. The local documentation can be found in `docsbuild/build/html`.
244120

245121
## License and Copyright
246122

0 commit comments

Comments
 (0)