Support creation of CDP DW Database Catalog and Virtual Warehouses #27

raju-saravanan · 2021-07-15T09:32:23Z

Changes in this PR:

Plugin support to create private load balancers for DW cluster.
Add support for the creation of DW database catalogs based on DW definitions.
Add support creation of DW virtual warehouses based on DW definitions.

Changes in this PR are done in conjunction with the following PRs:

Deploy : Add CDW playbook for DW datacatalog and virutal warehouse creation cloudera-deploy#46
Cloud :
- Expose modules to support creation of CDP DW Database Catalog and Virtual Warehouses cloudera.cloud#19 or
- Add and update CDW modules cloudera.cloud#29
CDPY:
- Support creation of CDP DW Database Catalog and Virtual Warehouses cdpy#23 and
- Update DW for improved error handling and VW creation cdpy#28

roles/runtime/tasks/initialize_base.yml

roles/runtime/tasks/initialize_setup_aws.yml

roles/runtime/defaults/main.yml

Chaffelson

Looks good. there's a minor hiccup with DW teardown but that is a bug already fixed in the purge PR, which is also due to be merged.

wmudge

Initial review. I'm going to continue testing and review, esp. with the changes/updates in the underlying cloudera.cloud.dw_* modules.

wmudge · 2021-08-20T14:49:04Z

roles/runtime/tasks/initialize_base.yml

+          name: "{{ item.1.name | default([run__namespace, run__dw_vw_suffix ,__dw_dbc_index] | join('-')) }}"
+          dbc_name: "{{ item.0.name }}"
+          use_default_dbc: "{{ item.0.use_default_dbc }}"
+          type: "{{ item.1.type | default('hive') }}"


This should be a variable set at the dw: global key level (with defaults)

wmudge · 2021-08-20T14:49:17Z

roles/runtime/tasks/initialize_base.yml

+          dbc_name: "{{ item.0.name }}"
+          use_default_dbc: "{{ item.0.use_default_dbc }}"
+          type: "{{ item.1.type | default('hive') }}"
+          template: "{{ item.1.template | default('xsmall') }}"


This should be a variable set at the dw: global key level (with defaults)

wmudge · 2021-08-20T14:49:40Z

roles/runtime/tasks/initialize_base.yml

+          use_default_dbc: "{{ item.0.use_default_dbc }}"
+          type: "{{ item.1.type | default('hive') }}"
+          template: "{{ item.1.template | default('xsmall') }}"
+          autoscaling_min_nodes: "{{ item.1.autoscaling.min_nodes | default(None) }}"


Better to use omit() than pass None or an empty config dict for lines 194-199

I am referencing these parameters in setup_aws file. So, if I use omit() here I would have to use omit() in setup_aws as well. If you are ok with this, I can make the changes

wmudge · 2021-08-20T14:52:37Z

roles/runtime/tasks/initialize_base.yml

+      loop: "{{ run__dw_dbc_configs | subelements('virtual_warehouses')}}"
+      loop_control:
+        index_var: __dw_dbc_index
+        label: "{{ item.0.name }}"


Should also have a loop_var to avoid using item

wmudge · 2021-08-20T14:53:40Z

roles/runtime/tasks/initialize_base.yml

+        run__dw_dbc_configs: "{{ run__dw_dbc_configs | default([]) | union([config]) }}"
+      vars:
+        include: "{{ lookup('template', __dw_config.include | default('experiences_config_placeholder.j2')) | from_yaml }}"
+        use_default_dbc: "{{ True if __dw_config.name is not defined else False | bool }}"


This can be inlined at line 177, also __dw_config.name is undefined | bool works?

wmudge · 2021-08-20T14:59:42Z

roles/runtime/tasks/initialize_setup_aws.yml

+    - name: Assign AWS Private Subnet IDs in not assinged
+      when: run__datahub_private_subnet_ids is undefined
+      ansible.builtin.set_fact:
+        run__datahub_private_subnet_ids: "{{ [] }}"


run__datahub_private_subnet_ids: "{{ run__datahub_private_subnet_ids | default([]) }}" and remove the when

wmudge · 2021-08-20T15:11:58Z

roles/runtime/tasks/setup_aws.yml

+    - name: Create CDP DW Database catalogs
+      when: not __dw_dbc_config.use_default_dbc
+      cloudera.cloud.dw_database_catalog:
+        cluster_id : "{{ run__dw_list.clusters[0].id }}"


Perhaps pull out into a separate task that can ensure the cluster 0 exists, etc. -- the set_fact variable could be __default_dbc etc.

wmudge · 2021-08-20T15:15:21Z

roles/runtime/tasks/setup_aws.yml

+    - name: Set CDP DW Database catalog name to id map
+      when: __dw_dbc_build_async.database_catalogs is defined
+      ansible.builtin.set_fact:
+        run__dw_dbc_ids: "{{ run__dw_dbc_ids | default({}) | combine({ __dw_dbc_build_async.database_catalogs[0].name : __dw_dbc_build_async.database_catalogs[0].id}) }}"


Output of dw_database_catalog now returns a dict directly, not a list.

wmudge · 2021-08-20T15:20:51Z

roles/runtime/tasks/setup_aws.yml

+    - name: Create CDP DW Virtual warehouse
+      cloudera.cloud.dw_virtual_warehouse:
+        cluster_id: "{{ run__dw_list.clusters[0].id }}"
+        dbc_id: "{{ run__dw_dbc_ids[__dw_vw_config.dbc_name] if not __dw_vw_config.use_default_dbc else run__dw_list.clusters[0].dbcs[0].id}}"


This can be streamlined via the __default_dbc etc., also could use ternary as well

wmudge · 2021-08-20T15:21:44Z

roles/runtime/tasks/setup_aws.yml

+        type: "{{ __dw_vw_config.type }}"
+        name: "{{ __dw_vw_config.name }}"
+        template: "{{ __dw_vw_config.template }}"
+        autoscaling_min_nodes: "{{ __dw_vw_config.autoscaling_min_nodes | int }}"


I think the omit() that would be set in initialize task list should carry through here to the module call.

Chaffelson · 2021-08-23T21:26:06Z

Create CDP DW Virtual warehouse is currently busted, and teardown is also not working. Please revise.

raju-saravanan · 2021-08-30T15:37:38Z

@Chaffelson: Rebased the PR and made changes to make it work with webster's PR for Cloudera cloud.

Chaffelson

One minor change

roles/runtime/tasks/setup_aws.yml

raju-saravanan · 2021-09-13T14:02:15Z

@wmudge : Can we get this PR merged ? Is there anything else we are waiting for ?

Chaffelson · 2021-09-13T14:05:10Z

@wmudge : Can we get this PR merged ? Is there anything else we are waiting for ?

Not to my knowledge, provided the .cloud and cdpy dependencies are also clear.

wmudge · 2021-09-13T16:38:19Z

@raju-saravanan Just wanted to get the other changes into main before tackling this. I will review again.

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

raju-saravanan · 2021-09-22T13:59:08Z

@wmudge: Added support for partial teardown. The user would have to declare 'dw.teardown' tag to delete a subset of database_catalogs or virtual_warehouse (this will not delete the cdw cluster itself), if this is not declared we will do a cascade delete of the entire cdw cluster. eg:

dw: 
   teardown:
      database_catalogs:
        - name: "sraju-custom-dbc-2"
        - id: "warehouse-1632239880-b896"
      virtual_warehouses:
        - id: "compute-1632311757-7b8g"
        - name: "srx-6-vw-1"
        - name: "srx-6-vw-0"
        - name: "srx-6-vw-4"

You can mention vw and dbc to be deleted with its id or name as shown above. Found a small bug in dw_database_catalog.py raised a pr for this here (cloudera-labs/cloudera.cloud#33), which needs to be merged before this PR.

Chaffelson · 2021-10-20T18:32:28Z

@wmudge Should this be closed as it is superseded by #57 ?

Chaffelson · 2022-11-02T10:35:48Z

Replaced by #102

raju-saravanan force-pushed the cdw-level-2 branch from a2026a1 to 0078f9d Compare July 15, 2021 09:38

Chaffelson reviewed Jul 15, 2021

View reviewed changes

roles/runtime/tasks/initialize_base.yml Outdated Show resolved Hide resolved

roles/runtime/tasks/initialize_base.yml Show resolved Hide resolved

roles/runtime/tasks/initialize_setup_aws.yml Outdated Show resolved Hide resolved

guruchai reviewed Jul 20, 2021

View reviewed changes

roles/runtime/defaults/main.yml Show resolved Hide resolved

raju-saravanan force-pushed the cdw-level-2 branch 2 times, most recently from 8217872 to b290255 Compare July 22, 2021 13:18

Chaffelson approved these changes Jul 22, 2021

View reviewed changes

raju-saravanan force-pushed the cdw-level-2 branch 3 times, most recently from a4f0c27 to 8bd5738 Compare July 27, 2021 17:18

raju-saravanan force-pushed the cdw-level-2 branch from 8bd5738 to 079ae61 Compare August 5, 2021 10:47

wmudge requested changes Aug 20, 2021

View reviewed changes

Chaffelson self-requested a review August 23, 2021 21:25

raju-saravanan force-pushed the cdw-level-2 branch 2 times, most recently from f98c32d to ea4323d Compare August 30, 2021 12:15

Chaffelson suggested changes Sep 8, 2021

View reviewed changes

roles/runtime/tasks/setup_aws.yml Show resolved Hide resolved

Chaffelson approved these changes Sep 10, 2021

View reviewed changes

wmudge added the enhancement MINOR - New feature or enhancement in the CHANGELOG label Sep 13, 2021

wmudge approved these changes Sep 16, 2021

View reviewed changes

wmudge mentioned this pull request Sep 16, 2021

Support CDW Database Catalogs and Virtual Warehouses #49

Closed

raju-saravanan requested review from Chaffelson and wmudge September 17, 2021 15:28

Support CDW VW changes

3d54286

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

raju-saravanan added 6 commits September 20, 2021 09:52

Revert unnecessary changes

a113082

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Add labels for the loops

d386477

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Remove use_default_dbc flag

8d08f52

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Change the vw and dbc module names and other minor refactor

b74ef0a

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Minor refactor

41c6383

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

Address review comments

84576b9

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

raju-saravanan force-pushed the cdw-level-2 branch from ea4323d to 84576b9 Compare September 20, 2021 20:48

Refactor cdw teardown logic

bf69eed

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

raju-saravanan force-pushed the cdw-level-2 branch from 3765202 to bf69eed Compare September 20, 2021 20:50

Add support for CDW partial teardown

ababc1e

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

raju-saravanan force-pushed the cdw-level-2 branch from 31bc9e7 to ababc1e Compare September 22, 2021 13:43

Remove logs from code

a85ea21

Signed-off-by: Saravanan Raju <saravanan.footloose@gmail.com>

raju-saravanan force-pushed the cdw-level-2 branch from 2cce0be to a85ea21 Compare September 22, 2021 13:45

wmudge mentioned this pull request Oct 6, 2021

Support for CDW Data Service #57

Closed

wmudge marked this pull request as draft April 1, 2022 16:29

Chaffelson closed this Nov 2, 2022

wmudge added skip_changelog SKIP - Do not include in the CHANGELOG and removed enhancement MINOR - New feature or enhancement in the CHANGELOG labels Jul 15, 2025

Support creation of CDP DW Database Catalog and Virtual Warehouses #27

Support creation of CDP DW Database Catalog and Virtual Warehouses #27

Uh oh!

Conversation

raju-saravanan commented Jul 15, 2021 • edited by wmudge Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chaffelson left a comment

Choose a reason for hiding this comment

Uh oh!

wmudge left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chaffelson commented Aug 23, 2021

Uh oh!

raju-saravanan commented Aug 30, 2021

Uh oh!

Chaffelson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raju-saravanan commented Sep 13, 2021

Uh oh!

Chaffelson commented Sep 13, 2021

Uh oh!

wmudge commented Sep 13, 2021

Uh oh!

raju-saravanan commented Sep 22, 2021

Uh oh!

Chaffelson commented Oct 20, 2021

Uh oh!

Chaffelson commented Nov 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

raju-saravanan commented Jul 15, 2021 •

edited by wmudge

Loading