CE-338 Master updation with latest code base as of 25/05/2023 (#43)

CE-337 Creation and updation of 'develop' branch with latest codebase (#42)

* DXE-683 Rename Select Connector - CouchBase

* Version 1.1.1 (#31)

* initial support for using non couchbase user through sudo

* issue #22 - initial deployment, index support changes for backup ingestion, various fixes

* partial work

* cbbackupmgr working with sudo, bucket list in snapshot in JSON

* version 1.0.2

* fixing uid/gid regex

* version 1.0.3 - error messages fix

* ver 1.0.3.1

* ver 1.0.3.2 - sudo for all

* resync of xdcr and backupmgr OK

* workinf with copy config

* working with indexes - tested with backup only - ee, analytics removed from schema

* add end of multi cluster

* working multinode

* checks added, upgrade added

* version 1.1.0 build 0

* check mount points / doc updates

* cbbackup exit code check

* xdcr fixed for multi IP, and different scenarios

* v1.1.1 build 1.1.0.2

* v1.1.0 build 1.1.0.2

* version 1.1.1 - added field for hostname

* Issue #32 - multinode start

* version 1.1.2

* Set theme jekyll-theme-minimal

* doc fixes

* new logo

* proper style

* docs with multi node




* DXE-988 Couchbase - VDB failed to enable (#36)

* CE-59 Couchbase Plugin : Conversion of python2 to python3 (#38)

* CE-332 Disable python2.7 unit tests on couchbase repository (#41)

* unit test upgrade to python3.8

PR URL: https://www.github.com/delphix/couchbase-plugin/pull/41

* unit test upgrade to python3.8

* removing python2.7 tests

* fixing python version to python3.8 for pytest

* disabling unit tests in github workflows

* reverting changes in test

---------

Co-authored-by: atyagi <ankur.tyagi@delphix.com>
Co-authored-by: Marcin Przepiorowski <pioro1@gmail.com>
Co-authored-by: Marcin Przepiorowski <marcin@delphix.com>

Author: 4 people

docs/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-minimal

docs/docs/.pages

@@ -1,7 +1,6 @@
 arrange:
   - Overview: 'index.md'
   - [Installation]: 'Installation.md'
-  - PluginDevelopment : 'PluginDevelopment.md'
   - Discovery: 'Discovery.md'
   - Linking: 'Linking.md'
   - Provisioning: 'Provisioning.md'

docs/docs/Discovery.md

@@ -5,33 +5,27 @@ Environment discovery is a process that enables the Couchbase Plugin to determin
 . Whenever there is any change (installing a new database home) to an already set up environment in the Delphix application, we need to perform an environment refresh. 
 
 
-Prerequisites
-=============
+## Prerequisites
 
--   A source environment must be added to the Delphix Engine.
+-   A stagine or target environment must be added to the Delphix Engine.
 -   Installation of the Couchbase Plugin is required before the Discovery. 
 -   Environment variable `$COUCHBASE_PATH ` should set on staging/target host, which contains the binary path of Couchbase.
 
 
-Refreshing an Environment
-=========================
-Environment refresh will update the metadata associated with that environment and send a new Plugin to the host.
+### Configuring a staging environment 
 
-1. Login to the **Delphix Management** application.
-2. Click **Manage**.
-3. Select **Environments**.
-4. In the Environments panel, click the name of the environment you want to refresh.
-5. Select the **Refresh** icon.
-6. In the Refresh confirmation dialog select **Refresh**.
+Environments contain repositories, and each environment may have any number of repositories associated with it.
+Couchbase is limiting number of Couchbase binaries installed on server to one, and in that case each environment will contain
+a single repository with current Couchbase installation.
+
+Repository contains database instances and in each repository any number of source cluster can be configured.
+Please keep in mind that only one dSource can be enabled simiultaniusly on a given staging server.
 
-![Screenshot](/couchbase-plugin/image/image9.png)
+For next step go to XDCR Setup or Backup Setup sections below
 
+### XDCR Setup
 
-XDCR Setup
-===================
-Environments exist to contain `repositories`, and each environment may have any number of repositories associated with it.
-`Repository` contains database instances and in each repository any number of `SourceConfig` objects, which represent known database instances. Source config is not generated automatically in
- Couchbase plugin. Therefore, we need to add `SourceConfig` object through which can create a dSource. 
+By default Couchbase cluster are not discovered and has to be added manually using the following steps:
 
 1. Login to the **Delphix Management** application.
 2. Click **Manage**.
@@ -40,40 +34,46 @@ Environments exist to contain `repositories`, and each environment may have any
 5. Click on **+** icon (Shown in next image).
 
 
-![Screenshot](/couchbase-plugin/image/image10.png)
+![Screenshot](./image/add_repository.png)
 
 
 6. Add required details in the `Add database` section.
  - Enter port number in **Source Couchbase port** section.
- - Enter source host address in section **Source Host**.
+ - Enter source host address in section **Source Host** ( this will be used as Source cluster for XDCR replication )
  - Enter unique name for the staging database in **identify field** section.
  - Enter Couchbase data path of staging host in **DB data path** section.
 
 
-![Screenshot](/couchbase-plugin/image/image11.png)
-
+![Screenshot](./image/add_sourceconfig_xdcr.png)
 
-CBBACKUPMGR Setup
-=================
+### Backup Setup
 
-The steps to add source config remain the same as we saw in XDCR setup. In this approach, we don't connect to source environment as this is zero-touch production approach.
-We can enter any random or dummy value in this field of source host name when we choose CBBACKUPMGR option for data ingestion.
+By default Couchbase clusters are not discovered and has to be added manually using the following steps:
 
 1. Login to the **Delphix Management** application.
 2. Click **Manage**.
 3. Select **Environments**.
 4. Select the repository.
-5. Click on **+** icon (Shown in next image).  
-![Screenshot](/couchbase-plugin/image/image10.png)
+5. Click on **+** icon (Shown in next image).
 
-6. In the **Add Database** section enter the following information:
- - `Source Couchbase port`: This is the port number to be used by Couchbase services.
- - `Source Host`: Leave this field as blank.
- - `identity field`: Provide unique name for staging database.
- - `DB data path`: Leave this field as blank.
+![Screenshot](./image/add_repository.png)
 
+6. Add required details in the `Add database` section.
+ - Enter port number in **Source Couchbase port** section ( this can be any dummy number - this field is not used for backup based ingestion )
+ - Enter source host address in section **Source Host** ( this can be any dummy number - this field is not used for backup based ingestion )
+ - Enter unique name for the staging database in **identify field** section.
+ - Enter Couchbase data path of staging host in **DB data path** section.
 
-![Screenshot](/couchbase-plugin/image/image11.png)
 
+## Refreshing an Environment
+Environment refresh will update the metadata associated with that environment and send a new Plugin to the host.
+
+1. Login to the **Delphix Management** application.
+2. Click **Manage**.
+3. Select **Environments**.
+4. In the Environments panel, click the name of the environment you want to refresh.
+5. Select the **Refresh** icon.
+6. In the Refresh confirmation dialog select **Refresh**.
 
+![Screenshot](./image/add_sourceconfig_backup.png)
 

docs/docs/Installation.md

@@ -14,22 +14,22 @@ Method1: Using GUI
 ------------------
 
 1. Click on **Manage** (present on top-left of this page) and then **Plugins**.  
-![Screenshot](/couchbase-plugin/image/image2.png)
+![Screenshot](./image/image2.png)
 
 2. Click on **+** icon.  
-![Screenshot](/couchbase-plugin/image/image3.png)
+![Screenshot](./image/image3.png)
 
 3. Click on **Upload** a plugin.  
-![Screenshot](/couchbase-plugin/image/image4.png)
+![Screenshot](./image/image4.png)
 
 4. Select the `build(artifacts.json)` from your device.   
-![Screenshot](/couchbase-plugin/image/image5.png)
+![Screenshot](./image/image5.png)
 
 5. Click on **close** button.  
-![Screenshot](/couchbase-plugin/image/image6.png)
+![Screenshot](./image/image6.png)
 
 6. See the plugin version in **Plugins** section.  
-![Screenshot](/couchbase-plugin/image/image7.png)
+![Screenshot](./image/image7.png)
 
 
 Method2: Using dvp command

docs/docs/Linking.md

@@ -4,90 +4,140 @@ Linking a data source will create a dSource object on the engine and allow Delph
 
 
 
-Prerequisites
-=============
+## Prerequisites
 
-Discovery and source config object should be created on the staging host before proceeding to link Couchbase dataset. 
-
-
-Creating dSource
-==============
+Staging environment created and source database configured in discovered repository
 
+## Creating dSource using XDCR
 
 1. Login to **Delphix Management** application.
 2. Click **Manage** >  **Datasets**.
 3. Select **Add dSource**.
 4. In the Add dSource wizard, select the Couchbase source configuration which is created on the staging host.
 5. Enter the Couchbase-specific parameters for your dSource configuration.
-6. Select the dSource approach from the drop-down (XDCR and Couchbase Backup Manager) available on dSource wizard.
+6. Select the dSource type XDCR from the drop-down available on dSource wizard.
 7. Based on approach selection, follow the steps either for XDCR or Couchbase Backup Manager method. The Description of both methods is below.
-
-
- Method1: XDCR
---------------
-Cross datacenter replication allows data to be replicated across clusters that are potentially located in different data centers.
-
-
-1. Enter the details for **Staging Couchbase host** - FQDN or IP address recommended.
-2. Enter the details for **Staging Port Number** available on the staging host. The default port for couchbase is 8091.
-3. Enter the details for **Mount Path** available on the staging host. This empty folder acts as a base for NFS mounts.
-4. Enter the details for **Staging Cluster Name** to setup new cluster on the staging host.
-5. Enter the configuration details for your staging cluster as per resource availability on the staging host.
+8. Enter the details for **Staging Couchbase host** - FQDN or IP address recommended.
+9. Enter the details for **Staging Port Number** available on the staging host. The default port for couchbase is 8091.
+10. Enter the details for **Mount Path** available on the staging host. This empty folder acts as a base for NFS mounts.
+11. Enter the details for **Staging Cluster Name** to setup new cluster on the staging host.
+12. Enter the configuration details for your staging cluster as per resource availability on the staging host.
     - Cluster RAM Size
     - Cluster Index RAM Size
     - Cluster FTS RAM Size
-    - Cluster Eventing RAM Size
-    - Cluster Analysis RAM Size              
-![Screenshot](/couchbase-plugin/image/image12.png)
+    - Cluster Eventing RAM Size - this should be 0
+    - Cluster Analysis RAM Size - this should be 0    
 
-6. Click on **+** plus symbol to modify configuration settings. Mention bucket list for which cross datacenter replication (XDCR) only be enabled.  
-![Screenshot](/couchbase-plugin/image/image14.png)
+    ![Screenshot](./image/add_dsource_1.png)
 
-7. Enter the details of **Bucket Name** to be part of XDCR. Then click on **Next** button  
-![Screenshot](/couchbase-plugin/image/image15.png)
+13. Enter the details for **Staging Cluster Admin User**  and **Staging Cluster Admin Password**
+14. Enter the details for **Source Cluster Admin User**  and **Source Cluster Admin Password**
 
-8. Provide the details for **dSource Name** and **Target group** on the dSource configuration page.  
-![Screenshot](/couchbase-plugin/image/image16.png)
+    ![Screenshot](./image/add_dsource_2.png)
 
-9. On the **Data management** page, select the following:
+15. If not all buckets needs to be replicated, click on **+** plus symbol to modify configuration settings. Mention bucket list for which cross datacenter replication (XDCR) only be enabled.  
+    ![Screenshot](./image/image14.png)
+
+16. Enter the details of **Bucket Name** to be part of XDCR. Then click on **Next** button  
+    ![Screenshot](./image/image15.png)
+
+17. Provide the details for **dSource Name** and **Target group** on the dSource configuration page.  
+    ![Screenshot](./image/add_dsource_3.png)
+
+18. On the **Data management** page, select the following:
     - Staging Environment: This will be your staging host where source config was created.
     - User: Database OS user with required privileges for linking the dataset.
-10. On the next section, review the configuration and click on **Next** button to view the summary.
-11. Click the **Submit** button which will initiate the linking process.
-![Screenshot](/couchbase-plugin/image/image17.png)
-12. Once dSource is created successfully, you can review the datasets on **Manage** > **Datasets** > **dSource Name**.  
-![Screenshot](/couchbase-plugin/image/image19.png)
 
+    ![Screenshot](./image/add_dsource_4.png)
+
+19. On the next screens, configure a policy, hooks and review the configuration and click on **Next** button to view the summary.
+
+    ![Screenshot](./image/add_dsource_5.png)
 
-Method2: Couchbase Backup Manager 
----------------------------------------
-**Note**: Follow the instructions below before creating dSource to avoid source/production server dependency.
+    ![Screenshot](./image/add_dsource_6.png)
 
-- Provide source server buckets related information in a file: */tmp/couchbase_src_bucket_info.cfg*.
-  `/opt/couchbase/bin/couchbase-cli bucket-list --cluster <sourcehost>:8091 --username $username --password $password`
 
-- **Backup Repository**: This file will be required at the time of dSource creation using CBBACKUPMGR.
-  `/opt/couchbase/bin/cbbackupmgr config --archive /u01/couchbase_backup --repo delphix`
+20. Click the **Submit** button which will initiate the linking process.
 
-- **Backup Location**: Get data from source host in backup directory of staging host.
-`/opt/couchbase/bin/cbbackupmgr backup -a /u01/couchbase_backup -r delphix -c couchbase://<hostname> -u user -p password`     
+    ![Screenshot](./image/add_dsource_7.png)
 
 
-**Procedure**:
+21. Once dSource is created successfully, you can review the datasets on **Manage** > **Datasets** > **dSource Name**.  
+
+
+     ![Screenshot](./image/dsource_ingested.png)
+
+
+## Creating dSource using backup
+
+Prerequisites:
+Access to production backup with those ex. values:
+
+/backup - archive
+
+PROD - repository
+
+```
+ /opt/couchbase/bin/cbbackupmgr info -a /backup -r PROD
+ Name  | Size     | # Backups  |
+ PROD  | 58.03MB  | 1          |
+ +  Backup                               | Size     | Type  | Source                              | Cluster UUID                      | Range  | Events  | Aliases  | Complete  |
+ +  2022-01-10T11_29_26.465860528-05_00  | 58.03MB  | FULL  | http://couchbasesrc.dlpxdc.co:8091  | 08f7937a26b2d20178a5ed16d7a2dd1c  | N/A    | 0       | 0        | true      |
+```
 
 
 1. Login to **Delphix Management** application.
 2. Click **Manage** >  **Datasets**.
 3. Select **Add dSource**.
-4. In the **Add dSource wizard**, select the Couchbase source configuration you created on the staging host.
+4. In the Add dSource wizard, select the Couchbase source configuration which is created on the staging host.
 5. Enter the Couchbase-specific parameters for your dSource configuration.
-6. Select the dSource type from the drop-down available on dSource wizard.
-7. When we select CBBACKUPMGR as dSource Type, the following fields on dSource wizard are mandatory.
-    - Enter the details for **Backup Location** where the backup files generated through CBBACKUPMGR are present on the staging host.
-    - Enter the details for **Backup Repository** that contains a backup configuration of staging host. 
-8. The remaining steps for CBBACKUPMGR ingestion are similar to XDCR. Use steps from the second point mentioned in XDCR method. 
+6. Select the dSource type Couchbase Backup Manager from the drop-down available on dSource wizard.
+7. Based on approach selection, follow the steps either for XDCR or Couchbase Backup Manager method. The Description of both methods is below.
+8. Enter the details for **Staging Couchbase host** - FQDN or IP address recommended.
+9. Enter the details for **Staging Port Number** available on the staging host. The default port for couchbase is 8091.
+10. Enter the details for **Backup Location** available on the staging host. 
+11. Enter the details for **Backup repository** 
+12. Enter the details for **Mount Path** available on the staging host. This empty folder acts as a base for NFS mounts.
+13. Enter the details for **Staging Cluster Name** to setup new cluster on the staging host.
+14. Enter the configuration details for your staging cluster as per resource availability on the staging host.
+    - Cluster RAM Size
+    - Cluster Index RAM Size
+    - Cluster FTS RAM Size
+    - Cluster Eventing RAM Size - this should be 0
+    - Cluster Analysis RAM Size - this should be 0    
+
+    ![Screenshot](./image/add_dsource_1backup.png)
+
+15. Enter the details for **Staging Cluster Admin User**  and **Staging Cluster Admin Password**
+16. Enter dummy values for **Source Cluster Admin User**  and **Source Cluster Admin Password** - they are not used
+
+    ![Screenshot](./image/add_dsource_2backup.png)
+
+17. Then click on **Next** button  
+
+17. Provide the details for **dSource Name** and **Target group** on the dSource configuration page.  
+    ![Screenshot](./image/add_dsource_3.png)
+
+18. On the **Data management** page, select the following:
+    - Staging Environment: This will be your staging host where source config was created.
+    - User: Database OS user with required privileges for linking the dataset.
+
+    ![Screenshot](./image/add_dsource_4.png)
+
+19. On the next screens, configure a policy, hooks and review the configuration and click on **Next** button to view the summary.
+
+    ![Screenshot](./image/add_dsource_5.png)
+
+    ![Screenshot](./image/add_dsource_6.png)
+
+
+20. Click the **Submit** button which will initiate the linking process.
+
+    ![Screenshot](./image/add_dsource_7backup.png)
+
+
+21. Once dSource is created successfully, you can review the datasets on **Manage** > **Datasets** > **dSource Name**.  
 
-Note: When we select dSource type as Couchbase Backup Manager, we do not require any details for the `Staging Couchbase Host` field.
 
-![Screenshot](/couchbase-plugin/image/image22.png)
+     ![Screenshot](./image/dsource_ingested.png)