THIS SERVICE HAS BEEN MERGED INTO ION-STORE https://github.com/connexta/ion-store

Ion-Ingest is now ARCHIVED

Ingest

Prerequisites

• Java 11
• Docker daemon

Working with IntelliJ or Eclipse

This repository uses Lombok, which requires additional configurations and plugins to work in IntelliJ / Eclipse. Follow the instructions here to set up your IDE.

Building

To just compile and build the projects:

./gradlew assemble

To do a full build with tests and the formatter:

./gradlew build

Build Checks

OWASP

./gradlew dependencyCheckAnalyze --info

The report for each project can be found at build/reports/dependency-check-report.html.

Style

The build can fail because the static analysis tool, Spotless, detects an issue. To correct most Spotless issues:

./gradlew spotlessApply

For more information about spotless checks see here.

Tests

• Tests are run automatically with ./gradlew build.
• To skip all tests, add -x test.
• Even if the tests fail, the artifacts are built and can be run.
• To change logging to better suit parallel builds pass -Pparallel or the --info flag
• To run a single test suite:

  ./gradlew test --tests TestClass

Integration Tests

• The integration tests require a Docker daemon.
• To skip integration tests, add -PskipITests.

Running

Configuring

1. The service can be configured with external configuration files that will be applied to the docker container during deployment. The configuration YAML files can be found at <PROJECT_ROOT>/configs/ and are not version-controlled. The properties in these files will be merged with any properties that you have configured in the service. The properties in the external config files take precedence over config files that are built with the service.

   Example configs/ingest_config.yml:

   endpointUrl:
     retrieve: http://localhost:9040/

   Example configs/s3_config.yml:

   s3:
     endpointUrl: http://minio:9000
     region: local
     bucket: metacard-quarantine

   Example configs/transform_config.yml:

   endpointUrl:
     transform: http://localhost:9090/transform/

   The service is capable of storing data in an S3-compatible data store. The configuration to access S3 is found as a list of commands under the store service in the docker-compose.yml file. Here you can change the endpoint URL, the S3 bucket name, and the credentials the service will use to connect to S3. The docker-compose.yml file uses docker secrets for the access key and secret key. The key values are stored in files called s3_access.sec and s3_secret.sec. These files must be in the same directory as the docker-compose.yml and are not version-controlled.

2. A Docker network named cdr is needed to run via docker-compose.

   Determine if the network already exists:

   docker network ls

   If the network exists, the output includes a reference to it:

   NETWORK ID          NAME                DRIVER              SCOPE
   zk0kg1knhd6g        cdr                 overlay             swarm

   If the network has not been created:

   docker network create --driver=overlay --attachable cdr

Running Locally via docker stack

docker stack deploy -c docker-compose.yml ingest-stack

Helpful docker stack Commands

• To stop the Docker service:

  docker stack rm ingest-stack

• To check the status of all services in the stack:

  docker stack services ingest-stack

• To stream the logs to the console for a specific service:

  docker service logs -f <service_name>

Running in the Cloud

There are two ways to configure the build system to deploy the service to a cloud:

• Edit the deploy.bash file. Set two variables near the top of the file:
  • SET_DOCKER_REG="ip:port"
  • SET_DOCKER_W="/path/to/docker/wrapper/"

OR

• Avoid editing a file in source control by exporting values:

  export DOCKER_REGISTRY="ip:port"
  export DOCKER_WRAPPER="/path/to/docker/wrapper/"

After configuring the build system:

./gradlew deploy

Inspecting

The service is deployed with (Springfox) Swagger UI. This library uses Spring Boot annotations to create documentation for the service endpoints. The /swagger-ui.html endpoint can be used to view Swagger UI. The service is also deployed with Spring Boot Actuator. The /actuator endpoint can be used to view the Actuator.