Add options for configuring custom deployments

[sandertan]

Hi CogStack team!

After doing a few local deployments of the CogStack Nifi project, we made some adjustments to make it easier to configure local parameters, and without running into merge conflicts with the master branch of this repo. It might be nice to also include these changes in master, so that other sites can benefit from this as well.

Changes

• Only track deploy/.env-example in Git and use deploy/.env for untracked local configurations.
• Add -example files for nifi.env en elasticsearch.env.
• Add documentation how to use these env files.
• Add environment variables for nifi.env and elasticsearch.env to deploy/services.yml and example files.
• Add environment variables for NiFi and ES security directories to deploy/services.yml and example files.
• Add documentation how to use local docker-compose files for local forks. This allows for tracking deployments similar to https://github.com/CogStack/MedCATtrainer, which contains multiple compose files.
• Add documentation how to use $COMPOSE_PROJECT_NAME, which can be useful when running multiple deployments on the same machine.

Wish list for other PRs

I've left out some other changes we made in our local deployment because it requires changes to parts of the deployment configuration we don't use, e.g. Tika and multiple MedCAT instances (we only use one). And I also wonder what you think about these suggestions, before I make this PR any more complex :)

Use ENV variables for all host ports

This makes it easier to configure ports without changing docker-compose files or services.yml. For example:

    ports:
      - ${KIBANA_HOST_PORT}:5601

In deploy/.env configure:

KIBANA_HOST_PORT=5601

Create example files for all other configs

For example for the elasticsearch .env and .yml files in security/.

Update and align MedCATService config

Update and align CogStack-NiFi/services/nlp-services/applications/medcat/config/env_app with https://github.com/CogStack/MedCATservice/blob/master/envs/env_medcat . The MedCATService config contains some old parameters for previous MedCAT versions, while some of the newer parameters are absent from the CogStack NiFi config.

Set NiFI HTTP(S) settings through ENV variables

Currently, the HTTP(S) settings are set in nifi.properties. Using environment variables to change these settings has no effect because env vars have a lower priority. So to change the configuration, a change to nifi properties is necessary and a new docker image has to be made. This is not a large task, but it is avoidable by setting the HTTP(S) settings in nifi properties to empty values and setting them with ENV variables, like explained in https://hub.docker.com/r/apache/nifi/

Other suggestions

• Have you seen nifi-registry (https://hub.docker.com/r/apache/nifi-registry)? It looks like a pretty nice solution for keeping track of flows. Currently not a priority for us.
• NiFi is currently running on 1.11.4, while the latest is 1.15.3. It might be nice to update. Also not a priority for us.

[tomolopolis]

Hey @sandertan this looks great. Will get this reviewed ASAP

Thanks!

[vladd-bit]

Great work. And about the NiFi update, it's currently on the dev branch, I will get it merged asap once all the changes are final & stable.

Will investigate nifi-registry, a bit weird that it's not part of the official repo.