CouchbaseFakeIt

Overview

couchbasefakeit is a Docker image designed for testing and local development. It starts up a single, standalone Couchbase Server instance within a Docker container and initializes it with buckets, indexes, and fake data that you define. Fake data is generated using FakeIt.

This can be very useful in reducing developer friction, providing a way to quickly and easily spin up a Couchbase server preinitialized for your application. By including an Dockerfile and associated configuration files within your source control repo, you can version your development data definitions along with your application.

Pulling

The latest version can be pulled using:

docker pull facetdigital/couchbasefakeit:latest

The latest tag will be the latest Enterprise edition of Couchbase, with the latest release of FakeIt.

Specific versions may also be available, such as enterprise-6.6.1. This would be the Enterprise edition of Couchbase, version 6.6.1.

Using couchbasefakeit

To use couchbasefakeit, create your own Dockerfile that uses couchbasefakeit as its base image. Then add configuration files to the /startup directory of the new image. You may also override environment variables to change the Couchbase Server configuration.

FROM facetdigital/couchbasefakeit:latest

# Customize environment
ENV CB_DATARAM=256 \
    CB_PASSWORD=mypassword

# Copy files
COPY . /startup/

# Create a new directory to hold the status of the node, i.e. initialized or not.
RUN mkdir /nodestatus
VOLUME /nodestatus

Alternatively, if you use docker-compose you would set it up like below:

version: "3.9"

services:
  couchbase:
    image: facetdigital/couchbasefakeit:enterprise-7.0.1
    environment:
      - CB_CLUSTER_NAME=my_cluster
      - CB_SERVICES=kv,n1ql,index,fts,eventing,cbas
      - CB_DATARAM=1024
      - CB_INDEXRAM=1024
      - CB_SEARCHRAM=1024
      - CB_ANALYTICSRAM=1024
      - CB_EVENTINGRAM=512
      - CB_INDEXSTORAGE=memory_optimized
    ports:
      - "8091-8096:8091-8096"
      - "11207:11207"
      - "11210-11211:11210-11211"
    expose:
      - "8091-8096"
    volumes:
      - startup:/startup
      - initStatus:/nodestatus
      - couchbase:/opt/couchbase/var

volumes:
  couchbase:
    name: couchbase
  startup:
    name: couchbase-startup
    driver_opts:
      type: none
      device: ./startup
      o: bind
  initStatus:
    name: initStatus

Environment Variables

The following environment variables can be set to change the Couchbase Server configuration:

Env Variable	Description
CB_CLUSTER_NAME	Specify the name of the cluster
CB_DATARAM	Data service RAM in megabytes, default 1024
CB_INDEXRAM	Index service RAM in megabytes, default 1024
CB_SEARCHRAM	Search (FTS) service RAM in megabytes, default 1024
CB_ANALYTICSRAM	Analytics service RAM in megabytes, default 1024. Only applicable if cbas is added to CB_SERVICES
CB_EVENTINGRAM	Eventing service RAM in megabytes, default 1024. Only applicable if eventing is added to CB_SERVICES
CB_SERVICES	Services to enable, default kv,n1ql,index,fts
CB_INDEXSTORAGE	Index storage mode, forestdb or memory_optimized (default)
CB_USERNAME	Couchbase user name, default Administrator
CB_PASSWORD	Couchbase password, default password

Values for CB_SERVICES and CB_INDEXSTORAGE correspond to parameters for the Couchbase REST API.

NOTE: If you configure CB_SERVICES to create the cbas analytics service, make sure you set CB_ANALYTICSRAM to a minimum of 1024.

Bucket Configuration

To configure your buckets, simply place a buckets.json file in the /startup directory of your image. This file should contain an array of bucket definition objects.

[
  {
    "name": "sample",
    "ramQuotaMB": 100,
    "bucketType": "couchbase",
    "authType": "sasl",
    "saslPassword": "",
    "evictionPolicy": "fullEviction",
    "replicaNumber": 0,
    "flushEnabled": 1,
    "loadData": 1
  },
  {
    "name": "default",
    "ramQuotaMB": 100,
    "bucketType": "couchbase",
    "authType": "sasl",
    "saslPassword": "",
    "evictionPolicy": "fullEviction",
    "replicaNumber": 0,
    "flushEnabled": 1,
    "loadData": 0
  }
]

Attribute names and values in this file correspond with the Couchbase REST API create bucket endpoint. That said, there is one attribute that is a custom attribute, which is the loadData attribute. Set this to 1 to load FakeIt data into Couchbase or to 0 to not load any data. If loadData is not present in the bucket configuration, data will be loaded.

If this file is not overridden in your image, it will create a single bucket named default with a RAM quota of 100MB.

Scopes and Collections

NOTE: Only applicable for Couchbase Server 7+

To create scopes and collections, create a file underneath /startup with the name of your bucket, and a file with the following name: collections.json. For example, /startup/sample/collections.json. Note Names are case sensitive.

The format of the collections.json file should be as follows:

{
  "scopes": {
    "your_scope_name": {
      "collections": [
        "your_collection_name_1",
        "your_collection_name_2",
        "your_collection_name_3"
      ]
    }
  }
}

The values to replace are your_scope_name and the values in the collections array. Note: You can use the _default scope if you'd like by replacing your_scope_name with _default. The _default scope is automatically created in all buckets and cannot be deleted. You can add multiple scopes each having their own collections. For example:

{
  "scopes": {
    "_default": {
      "collections": [
        "default_collection_name_1",
        "default_collection_name_2"
      ]
    },
    "my_scope": {
      "collections": [
        "default_collection_name_1",
        "default_collection_name_2"
      ]
    }
  }
}

RBAC Configuration

To configure RBAC users for Couchbase Server versions 5+, simply place a rbac-users.json file in the /startup directory of your image. This file should be an array of JSON objects that define the various users and roles that need to be associated with each user. See the following example on how to structure the file:

[
 {
   "rbacName": "App User",
   "rbacUsername": "app-user",
   "rbacPassword": "password",
   "roles": [
     "bucket_full_access[sample]"
   ]
 },
 {
   ...
 }
]

Information on the available roles can be found here. If you want to limit the role to a specific bucket, place the bucket name in brackets at the end of the name, i.e. bucket_full_access[sample].

Generating Data With FakeIt

To generate data with FakeIt, create a directory underneath /startup with the name of your bucket, and directory beneath that named models. For example, /startup/sample/models. Note that the names are case sensitive. Add your FakeIt YAML models to the models directory.

FakeIt will be run using these models automatically during startup. You may also include inputs, such as CSV files, in the image to be referenced by the models.

This process will be run before indexes are created so that index updates don't degrade the performance of the data inserts.

FakeIt usage with Couchbase 7+

When using Couchbase 7 or greater, data generated by FakeIt will be inserted into the _default scope and _default collection. If you would like to insert the data into a different scope and collection you must have created that scope and collection in Couchbase. See the Scopes and Collections section above on automating the creation of scopes and collections. Additionally, you must add the name of the scope and the collection you want to insert the data into within the model definiton.

For example, we have configured the countries.yaml and users.yaml model definitions in the example/sample/models directory to illustrate how to define the scope and collection to use or just the collection to use.

• The contacts model will be located within the _default scope in the _default collection because neither the scope or collection have been defined.
• The countries model will be located within the _default scope in the collection named collection_1 because only the collection was defined.
• The users model will be located within the my_scope scope in the users collection because the scope and collection were defined.

Creating Views

To create views, add a directory underneath /startup with the name of your bucket and a text file named views.json. This file should be a JSON object with one or more design document specifications. The name of each attribute should be the name of the design document.

{
  "customers": {
    "views": {
        "CustomersByFirstName": {
            "map": "function (doc, meta) {\n  if ((doc.type === \"customer\") && doc.firstName) {\n    emit(doc.firstName, null);\n  }\n}"
        }
    }
  }
}

Examples of the syntax for design documents can be found in the Couchbase documentation. Note that views.json has an extra nesting level above the Couchbase examples, as it supports more than one design document in a single file.

Creating Indexes

To create indexes, add a directory underneath /startup with the name of your bucket and a text file named indexes.n1ql. For example, /startup/default/indexes.n1ql. Note that the names are case sensitive.

Within this file, you can define the CREATE INDEX statements for your bucket, separated by semicolons. It is recommended for performance to use WITH {"defer_build": true} for all indexes, and use a BUILD INDEX statement at the end of the file.

CREATE PRIMARY INDEX `#primary` ON default WITH {"defer_build": true};
CREATE INDEX `Types` ON default (`type`) WITH {"defer_build": true};
BUILD INDEX ON default (`#primary`, `Types`);

Creating Indexes with YAML or JSON

Alternatively, you may add YAML or JSON files with index definitions under the /startup/<bucketname>/indexes folder. This operation uses couchbase-index-manager to create the indexes. See here for an explanation of the file format needed when using YAML or JSON files.

Analytics Dataset Setup

To setup the analytics service datasets, add a directory under the /startup/<bucketname>/analytics folder. Within this folder create a text file named dataset.json. For example, /startup/default/analytics/dataset.json. Note that the names are case sensitive.

Within this file, create a key called statements with an array as the value. Within the array you can define your CREATE DATAVERSE and CREATE DATASET statements for your bucket, separated by semicolons. Below is an example showing what the dataset.json structure should look like. IMPORTANT Make sure to append the proper USE statement to each CREATE DATASET statement so that it's placed in the proper DATAVERSE. Additionaly, always end your file with a CONNECT LINK Local; statement.

{
  "statements": [
    "CREATE DATAVERSE `sample` IF NOT EXISTS;",
    "USE `sample`; CREATE DATASET IF NOT EXISTS users ON `sample` WHERE `type` = 'user';",
    "USE `sample`; CONNECT LINK Local;"
  ]
}

Creating Analytics Indexes

To create analytics indexes, add a directory under the /startup/<bucketname>/analytics folder. Within this folder create a text file named indexes.json. For example, /startup/default/analytics/indexes.json. Note that the names are case sensitive.

Within this file, create a key called statements with an array as the value. Within the array you can define your CREATE INDEX statements for your DATASET, separated by semicolons. IMPORTANT Make sure your queries always start with a USE statment otherwise the query engine will have no idea which DATAVERSE to associate the index with.

{
  "statements": [
    "USE `sample`; CREATE INDEX `idx_users` IF NOT EXISTS ON `users` (id: string);"
  ]
}

Creating Full Text Search Indexes

To create FTS indexes, add a directory underneath /startup with the name of your bucket, and underneath that a fts directory. Within that, add a json file for each index, with the file name being the index name. For example, /startup/default/fts/my_index.json. Note that names are case sensitive.

Within this file, place the JSON index definition. This can be easily exported from the Couchbase Console.

To create FTS index aliases, add an additional file in the same folder named aliases.json. This file should be an object with each attribute being an alias name, and the value being an array of index names.

{
    "my_alias": ["my_index"],
    "my_second_alias": ["my_index_2", "my_index_3"]
}

Creating Events

Couchbase Eventing allows document mutations from a bucket to be streamed, processed using Javascript, and outputs performed such as storing new documents in another bucket. CouchbaseFakeIt can create and deploy these events automatically on startup.

First, add a directory within your startup folder named events. Within this directory, add two files for each event you'd like to deploy.

event-name.json will have configuration, specifically the depcfg configuration for source buckets, metadata buckets, and buckets which may be referenced by the event. The settings attribute can also be used to override any settings, defaults apply for any excluded settings. An easy way to get these settings is to manually configure an event and then get the definition from http://localhost:8096/api/v1/functions (use HTTP Basic Authentication).

{
  "depcfg": {
    "buckets": [
      {
        "alias": "dst",
        "bucket_name": "default",
        "access": "rw"
      }
    ],
    "curl": [],
    "metadata_bucket": "default",
    "source_bucket": "sample"
  },
  "settings": {
    "worker_count": 3
  }
}

The second file should have the same name but a .js extension, event-name.js. This file contains the Javascript for the event.

function OnUpdate(doc, meta) {
  // This example event copies all documents from the "example" bucket to the "default" bucket

  dst[meta.id] = doc;
}

function OnDelete(meta) {
}

The event will be automatically deployed once it is created with the "Everything" feed boundary. This means that all documents in the source bucket should be processed by the event at startup, including any documents created from models. However, the /nodestatus/initialized file will be created before all documents are processed, as events are asynchronous in nature.

Also, note that by default logging will be set to the DEBUG level and each event will use only 1 worker. This is because CouchbaseFakeIt is intended for local machine development or CI testing. These settings can be overridden in the JSON file in the settings attribute.

Custom Scripts

If you would like to run custom scripts, for example to perform additional setup that isn't part of CouchbaseFakeit, you can do so via the following steps:

1. In the /startup directory of your image, add a new folder called custom-scripts
2. Now add your custom bash scripts to the custom-scripts directory. i.e. /startup/custom-scripts/my-custom-script.sh

You can add multiple bash scripts to this folder. Each script will be processed sequentially. If an error occurs, the script will exit and no other scripts will be processed. Please pay attention to the logging output to be sure all your scripts complete successfully. These scripts will be processed after all other setup tasks have been completed.

Example

An example image configuration can be found here.

To run the example locally:

1. Ensure that Couchbase Server is not currently running on your machine to avoid port conflicts

2. git clone https://github.com/brantburnett/couchbasefakeit.git

3. cd couchbasefakeit/example

   • NOTE: if you are targeting Couchbase Server 7+, you'll want to perform the following steps before moving to step 4.

   • Open up the example/sample/models/countries.yaml file and uncomment the following line:

     # collection: collection_1

   • Open up the example/sample/models/users.yaml file and uncomment the following lines:

     # scope: my_scope
     # collection: users

4. docker-compose up -d

5. The server will be accessible at http://localhost:8091 after 15-30 seconds. The username is "Administrator", password is "password".

To shut down and cleanup:

1. docker-compose down

For more detailed examples of FakeIt models, see https://github.com/bentonam/fakeit/tree/dev/test/fixtures/models.