feat(mojaloop/#2092): upgrade nodeJS version for core services (#902)

feat(mojaloop/#2092): upgrade nodeJS version for core services - mojaloop/project#2092 - standardised CI scripts, however, the test-integration CI job needed minor tweaks to work with the central-ledger's implementation - updated docker-compose files to work correctly - added wait-4-docker npm script - fixed lint issues - updated .nvmrc to latest LTS version - added standard CI scripts/config to package.json: release, snapshot, standard-version, etc - updated gitignore to include test/results and IGNORE patterns - updated README with standard auto-release information - Fixed imports - Cleaned up Package.json - Removed unnecessary dependencies - Removed unnecessary files - Re-factored some integration test `wrapWithRetries` functions to cater for the expected return value, this will mean that integration tests should be more consistent and be more resilient to time-processing issues. - Added a slight bump to the `wrapWithRetries` test helper function's default params for retries, and timeout Notes: - 'get-port' dependency version is fixed to v5.1.1, this is because v9+ only supports ESM loaders and not CJS. This will need to be addressed in a future story. - Helm chart mountPaths need to be updated from `/opt/central-ledger` to `/opt/app` as follows: ```YAML volumeMounts: - mountPath: /opt/app/config name: <deployment-name> ``` BREAKING CHANGE: Major version bump for node v16 LTS support, re-structuring of project directories to align to core Mojaloop repositories and docker image now uses `/opt/app` instead of `/opt/central-ledger` which will impact config mounts.
mojaloop · May 26, 2022 · defff30 · defff30
1 parent c283a70
commit defff30
Show file tree

Hide file tree

Showing 21 changed files with 20,128 additions and 4,825 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
diff --git a/.nvmrc b/.nvmrc
@@ -1 +1 @@
-12.16.0
+16.15.0
diff --git a/Dockerfile b/Dockerfile
@@ -1,23 +1,24 @@
-FROM node:12.16.1-alpine as builder
-WORKDIR /opt/central-ledger
+FROM node:16.15.0-alpine as builder
+WORKDIR /opt/app
 
-RUN apk add --no-cache -t build-dependencies git make gcc g++ python libtool autoconf automake \
+RUN apk --no-cache add git
+RUN apk add --no-cache -t build-dependencies make gcc g++ python3 libtool libressl-dev openssl-dev autoconf automake \
     && cd $(npm root -g)/npm \
     && npm config set unsafe-perm true \
     && npm install -g node-gyp
 
-COPY package.json package-lock.json* /opt/central-ledger/
+COPY package.json package-lock.json* /opt/app/
 
 RUN npm install
 
-COPY src /opt/central-ledger/src
-COPY config /opt/central-ledger/config
-COPY migrations /opt/central-ledger/migrations
-COPY seeds /opt/central-ledger/seeds
-COPY test /opt/central-ledger/test
+COPY src /opt/app/src
+COPY config /opt/app/config
+COPY migrations /opt/app/migrations
+COPY seeds /opt/app/seeds
+COPY test /opt/app/test
 
-FROM node:12.16.1-alpine
-WORKDIR /opt/central-ledger
+FROM node:16.15.0-alpine
+WORKDIR /opt/app
 
 # Create empty log file & link stdout to the application log file
 RUN mkdir ./logs && touch ./logs/combined.log
@@ -27,7 +28,7 @@ RUN ln -sf /dev/stdout ./logs/combined.log
 RUN adduser -D ml-user 
 USER ml-user
 
-COPY --chown=ml-user --from=builder /opt/central-ledger .
+COPY --chown=ml-user --from=builder /opt/app .
 RUN npm prune --production
 
 EXPOSE 3001

diff --git a/README.md b/README.md
@@ -1,4 +1,5 @@
 # central-ledger
+
 [![Git Commit](https://img.shields.io/github/last-commit/mojaloop/central-ledger.svg?style=flat)](https://github.com/mojaloop/central-ledger/commits/master)
 [![Git Releases](https://img.shields.io/github/release/mojaloop/central-ledger.svg?style=flat)](https://github.com/mojaloop/central-ledger/releases)
 [![Docker pulls](https://img.shields.io/docker/pulls/mojaloop/central-ledger.svg?style=flat)](https://hub.docker.com/r/mojaloop/central-ledger)
@@ -12,24 +13,29 @@ The central ledger is a series of services that facilitate clearing and settleme
 
 The following documentation represents the services, APIs and endpoints responsible for various ledger functions.
 
-## Contents:
-
-- [Running Locally](#running-locally)
-- [Configuration](#configuration)
-- [API](#api)
-- [Logging](#logging)
-- [Tests](#tests)
-- [Auditing Dependencies](#auditing-dependencies)
-- [Container Scans](#container-scans)
+## Contents
+
+- [central-ledger](#central-ledger)
+  - [Contents](#contents)
+  - [Running Locally](#running-locally)
+  - [Configuration](#configuration)
+    - [Environment variables](#environment-variables)
+  - [API](#api)
+  - [Logging](#logging)
+  - [Tests](#tests)
+    - [Running Integration Tests interactively](#running-integration-tests-interactively)
+  - [Container Scans](#container-scans)
+  - [Automated Releases](#automated-releases)
+    - [Potential problems](#potential-problems)
 
 ## Running Locally
 
 Please follow the instruction in [Onboarding Document](Onboarding.md) to setup and run the service locally.
 
-
 ## Configuration
 
 ### Environment variables
+
 The Central Ledger has many options that can be configured through environment variables.
 
 | Environment variable | Description | Example values |
@@ -49,7 +55,6 @@ The Central Ledger has many options that can be configured through environment v
 | CLEDG\_AMOUNT__PRECISION | Numeric value used to determine precision recorded for transfer amounts on this ledger. | 10 |
 | CLEDG\_AMOUNT__SCALE | Numeric value used to determine scale recorded for transfer amounts on this ledger. | 2 |
 
-
 ## API
 
 For endpoint documentation, see the [API documentation](API.md).
@@ -74,35 +79,56 @@ Tests include code coverage via istanbul. See the test/ folder for testing scrip
 
 ### Running Integration Tests interactively
 
-If you want to run integration tests in a repetitive manner, you can startup the test containers using `docker-compose`, login to running `central-ledger` container like so:
+If you want to run integration tests in a repetitive manner, you can startup the test containers using `docker-compose` via one of the following methods:
+
+- Running locally
+
+    Start containers required for Integration Tests
+
+    ```bash
+    docker-compose -f docker-compose.yml up -d kafka mysql
+    ```
+
+    Run wait script which will report once all required containers are up and running
+
+    ```bash
+    npm run wait-4-docker
+    ```
+
+    Run the Integration Tests
+
+    ```bash
+    npm run test:int
+    ```
 
-```bash
-docker-compose -f docker-compose.yml -f docker-compose.integration.yml up -d kafka mysql central-ledger
-```
 - Running inside docker
-  ```bash
-  docker exec -it cl_central-ledger sh
-  export CL_DATABASE_HOST=mysql
-  npm run migrate #first time only
-  npm run test:int
-  ```
-- Running natively
-  ```bash
-  export CL_DATABASE_HOST=localhost
-  npm run migrate #first time only
-  npm run test:int
+
+    Start containers required for Integration Tests, including a `central-ledger` container which will be used as a proxy shell.
+
+    ```bash
+    docker-compose -f docker-compose.yml -f docker-compose.integration.yml up -d kafka mysql central-ledger
+    ```
+
+    Run the Integration Tests from the `central-ledger` container
+
+    ```bash
+    docker exec -it cl_central-ledger sh
+    export CL_DATABASE_HOST=mysql
+    npm run test:int
   ```
 
 ## Auditing Dependencies
 
 We use `npm-audit-resolver` along with `npm audit` to check dependencies for node vulnerabilities, and keep track of resolved dependencies with an `audit-resolve.json` file.
 
 To start a new resolution process, run:
+
 ```bash
 npm run audit:resolve
 ```
 
 You can then check to see if the CI will pass based on the current dependencies with:
+
 ```bash
 npm run audit:check
 ```
@@ -119,3 +145,32 @@ For more information on anchore and anchore-cli, refer to:
 - [Anchore CLI](https://github.com/anchore/anchore-cli)
 - [Circle Orb Registry](https://circleci.com/orbs/registry/orb/anchore/anchore-engine)
 
+## Automated Releases
+
+As part of our CI/CD process, we use a combination of CircleCI, standard-version
+npm package and github-release CircleCI orb to automatically trigger our releases
+and image builds. This process essentially mimics a manual tag and release.
+
+On a merge to master, CircleCI is configured to use the mojaloopci github account
+to push the latest generated CHANGELOG and package version number.
+
+Once those changes are pushed, CircleCI will pull the updated master, tag and
+push a release triggering another subsequent build that also publishes a docker image.
+
+### Potential problems
+
+- There is a case where the merge to master workflow will resolve successfully, triggering
+  a release. Then that tagged release workflow subsequently failing due to the image scan,
+  audit check, vulnerability check or other "live" checks.
+
+  This will leave master without an associated published build. Fixes that require
+  a new merge will essentially cause a skip in version number or require a clean up
+  of the master branch to the commit before the CHANGELOG and bump.
+
+  This may be resolved by relying solely on the previous checks of the
+  merge to master workflow to assume that our tagged release is of sound quality.
+  We are still mulling over this solution since catching bugs/vulnerabilities/etc earlier
+  is a boon.
+
+- It is unknown if a race condition might occur with multiple merges with master in
+  quick succession, but this is a suspected edge case.
diff --git a/audit-resolve.json b/audit-resolve.json
@@ -1,96 +1,94 @@
 {
   "decisions": {
-    "1064661|@mojaloop/central-services-shared>shins>markdown-it": {
+    "1070030|@mojaloop/central-services-shared>shins>markdown-it": {
       "decision": "ignore",
-      "madeAt": 1648636266226,
-      "expiresAt": 1651228244345
+      "madeAt": 1653393759905,
+      "expiresAt": 1655985751719
     },
-    "1064761|@mojaloop/central-services-shared>widdershins>swagger2openapi>better-ajv-errors>jsonpointer": {
+    "1070030|widdershins>markdown-it": {
       "decision": "ignore",
-      "madeAt": 1648636267775,
-      "expiresAt": 1651228244345
+      "madeAt": 1653393759905,
+      "expiresAt": 1655985751719
     },
-    "1064761|@mojaloop/central-services-shared>widdershins>swagger2openapi>oas-validator>better-ajv-errors>jsonpointer": {
+    "1068155|@mojaloop/central-services-shared>shins>markdown-it>sanitize-html": {
       "decision": "ignore",
-      "madeAt": 1648636267775,
-      "expiresAt": 1651228244345
+      "madeAt": 1653393760886,
+      "expiresAt": 1655985751719
     },
-    "1064768|hapi-swagger>swagger-parser>z-schema>validator": {
+    "1070260|@mojaloop/central-services-shared>shins>markdown-it>sanitize-html": {
       "decision": "ignore",
-      "madeAt": 1648636268840,
-      "expiresAt": 1651228244345
+      "madeAt": 1653393762162,
+      "expiresAt": 1655985751719
     },
-    "1065159|@mojaloop/central-services-shared>widdershins>swagger2openapi>oas-validator>ajv": {
+    "1070256|ejs": {
       "decision": "ignore",
-      "madeAt": 1648636270152,
-      "expiresAt": 1651228244345
+      "madeAt": 1653393763207,
+      "expiresAt": 1655985751719
     },
-    "1065367|@mojaloop/central-services-shared>shins>sanitize-html": {
+    "1068386|hapi-auth-basic>hapi": {
       "decision": "ignore",
-      "madeAt": 1648636271777,
-      "expiresAt": 1651228244345
+      "madeAt": 1653393764193,
+      "expiresAt": 1655985751719
     },
-    "1065368|@mojaloop/central-services-shared>shins>sanitize-html": {
+    "1068399|hapi-auth-basic>hapi>ammo": {
       "decision": "ignore",
-      "madeAt": 1648636271777,
-      "expiresAt": 1651228244345
+      "madeAt": 1653393768017,
+      "expiresAt": 1655985751719
     },
-    "1065523|@mojaloop/central-services-shared>widdershins>yargs>yargs-parser": {
+    "1068389|hapi-auth-basic>hapi>ammo>subtext": {
       "decision": "ignore",
-      "madeAt": 1648636274889,
-      "expiresAt": 1651228244345
-    },
-    "1069987|@mojaloop/central-services-shared>widdershins>urijs": {
-      "decision": "fix",
-      "madeAt": 1652705152224
-    },
-    "1069988|@mojaloop/central-services-shared>widdershins>urijs": {
-      "decision": "fix",
-      "madeAt": 1652705152224
-    },
-    "1067553|@mojaloop/central-services-shared>widdershins>swagger2openapi>better-ajv-errors>jsonpointer": {
-      "decision": "postpone",
-      "madeAt": 1652795995442
+      "madeAt": 1653393769101,
+      "expiresAt": 1655985751719
     },
-    "1067553|@mojaloop/central-services-shared>widdershins>swagger2openapi>oas-validator>better-ajv-errors>jsonpointer": {
-      "decision": "postpone",
-      "madeAt": 1652795995442
+    "1068390|hapi-auth-basic>hapi>ammo>subtext": {
+      "decision": "ignore",
+      "madeAt": 1653393770290,
+      "expiresAt": 1655985751719
     },
     "1067560|hapi-swagger>swagger-parser>z-schema>validator": {
-      "decision": "postpone",
-      "madeAt": 1652795996027
+      "decision": "ignore",
+      "madeAt": 1653393771428,
+      "expiresAt": 1655985751719
     },
-    "1067946|@mojaloop/central-services-shared>widdershins>swagger2openapi>oas-validator>ajv": {
-      "decision": "postpone",
-      "madeAt": 1652795996489
+    "1067553|swagger2openapi>better-ajv-errors>jsonpointer": {
+      "decision": "ignore",
+      "madeAt": 1653393772614,
+      "expiresAt": 1655985751719
     },
-    "1068155|@mojaloop/central-services-shared>shins>sanitize-html": {
-      "decision": "postpone",
-      "madeAt": 1652795997039
+    "1067946|swagger2openapi>better-ajv-errors>jsonpointer>oas-validator>ajv": {
+      "decision": "ignore",
+      "madeAt": 1653393773753,
+      "expiresAt": 1655985751719
     },
-    "1070260|@mojaloop/central-services-shared>shins>sanitize-html": {
-      "decision": "postpone",
-      "madeAt": 1652795997039
+    "1068310|widdershins>markdown-it>yargs>yargs-parser": {
+      "decision": "ignore",
+      "madeAt": 1653393774821,
+      "expiresAt": 1655985751719
     },
-    "1068310|@mojaloop/central-services-shared>widdershins>yargs>yargs-parser": {
-      "decision": "postpone",
-      "madeAt": 1652795997537
+    "1070030|@mojaloop/central-services-shared>widdershins>markdown-it": {
+      "decision": "ignore",
+      "madeAt": 1653397119849,
+      "expiresAt": 1655989102215
     },
-    "1070030|@mojaloop/central-services-shared>shins>markdown-it": {
-      "decision": "postpone",
-      "madeAt": 1652795998107
+    "1070030|shins>markdown-it": {
+      "decision": "ignore",
+      "madeAt": 1653397119849,
+      "expiresAt": 1655989102215
     },
-    "1070207|@mojaloop/central-services-stream>async": {
-      "decision": "postpone",
-      "madeAt": 1652795998636
+    "1068310|@mojaloop/central-services-shared>widdershins>markdown-it>yargs>yargs-parser": {
+      "decision": "ignore",
+      "madeAt": 1653397120922,
+      "expiresAt": 1655989102215
     },
-    "1070245|@mojaloop/event-sdk>moment": {
-      "decision": "postpone",
-      "madeAt": 1652795999156
+    "1068155|shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1653397121987,
+      "expiresAt": 1655989102215
     },
-    "1070256|@mojaloop/central-services-shared>shins>ejs": {
-      "decision": "postpone",
-      "madeAt": 1652795999627
+    "1070260|shins>markdown-it>sanitize-html": {
+      "decision": "ignore",
+      "madeAt": 1653397123215,
+      "expiresAt": 1655989102215
     }
   },
   "rules": {},

diff --git a/docker-compose.integration.yml b/docker-compose.integration.yml
@@ -15,10 +15,10 @@ services:
     ports:
       - "3001:3001"
     volumes:
-      - ./docker/central-ledger/default.json:/opt/central-ledger/config/default.json
-      - ./test:/opt/central-ledger/test
-      - ./src:/opt/central-ledger/src
-      # - ./node_modules:/opt/central-ledger/node_modules
+      - ./docker/central-ledger/default.json:/opt/app/config/default.json
+      - ./test:/opt/app/test
+      - ./src:/opt/app/src
+      # - ./node_modules:/opt/app/node_modules
     environment:
       - CLEDG_MONGODB__DISABLED=true
       # Disable audit logs - makes it too verbose