es7-persistence module migration from 2.31.4 to 3.0.6 (Netflix#2324)

* es7-persistence module migration from 2.31.4 to 3.0.6

* revert conductor dependencies.lock

* finalize es7 migration

* update es-7 README.md

* update es-7 README

* rollback conductor es-7 properties migration

* fix:Create indexes on start-up including task_log one

* include es7-persistence to project

* shade es7-persistence

* fix es7 README.md

* replace deprecated method

* set ext['elasticsearch.version'] explicitly to avoid conflicts

* update es7-persistence README

* fix revElasticSearch7 version, shaded jar, add reminding note

* fix elasticsearch container version

* remove revElasticSearch7 from build.gradle

* remove not necessary ext['elasticsearch.version'], replace _index_template to _index

Co-authored-by: Marco Patricio Crasso <marco.crasso@invitae.com>

CHANGELOG.md

@@ -195,6 +195,29 @@ Changes to configurations:
 | workflow.elasticsearch.auto.index.management.enabled | conductor.elasticsearch.autoIndexManagementEnabled | true |
 | workflow.elasticsearch.document.type.override | conductor.elasticsearch.documentTypeOverride | "" |
 
+`es7-persistence` module:
+
+| Old | New | Default |
+| --- | --- | --- |
+| workflow.elasticsearch.version | conductor.elasticsearch.version | 7 |
+| workflow.elasticsearch.url | conductor.elasticsearch.url | localhost:9300 |
+| workflow.elasticsearch.index.name | conductor.elasticsearch.indexPrefix | conductor |
+| workflow.elasticsearch.tasklog.index.name | _removed_ |  |
+| workflow.elasticsearch.cluster.health.color | conductor.elasticsearch.clusterHealthColor | green |
+| workflow.elasticsearch.archive.search.batchSize | _removed_ |  |
+| workflow.elasticsearch.index.batchSize | conductor.elasticsearch.indexBatchSize | 1 |
+| workflow.elasticsearch.async.dao.worker.queue.size | conductor.elasticsearch.asyncWorkerQueueSize | 100 |
+| workflow.elasticsearch.async.dao.max.pool.size | conductor.elasticsearch.asyncMaxPoolSize | 12 |
+| workflow.elasticsearch.async.buffer.flush.timeout.seconds | conductor.elasticsearch.asyncBufferFlushTimeout | 10s |
+| workflow.elasticsearch.index.shard.count | conductor.elasticsearch.indexShardCount | 5 |
+| workflow.elasticsearch.index.replicas.count | conductor.elasticsearch.indexReplicasCount | 1 |
+| tasklog.elasticsearch.query.size | conductor.elasticsearch.taskLogResultLimit | 10 |
+| workflow.elasticsearch.rest.client.connectionRequestTimeout.milliseconds | conductor.elasticsearch.restClientConnectionRequestTimeout | -1 |
+| workflow.elasticsearch.auto.index.management.enabled | conductor.elasticsearch.autoIndexManagementEnabled | true |
+| workflow.elasticsearch.document.type.override | conductor.elasticsearch.documentTypeOverride | "" |
+| workflow.elasticsearch.basic.auth.username | conductor.elasticsearch.username | "" |
+| workflow.elasticsearch.basic.auth.password | conductor.elasticsearch.password | "" |
+
 `grpc-server` module:
 
 | Old | New | Default |

build.gradle

@@ -7,6 +7,7 @@ buildscript {
     }
     dependencies {
         classpath 'com.netflix.nebula:gradle-extra-configurations-plugin:5.0.3'
+        // revElasticSearch7 in dependencies.gradle needs to be updated when spring is upgraded
         classpath 'org.springframework.boot:spring-boot-gradle-plugin:2.3.12.RELEASE'
     }
 }

dependencies.gradle

@@ -28,6 +28,7 @@ ext {
     revCuratorTest = '2.4.0'
     revDynoQueues = '2.0.20'
     revElasticSearch6 = '6.8.12'
+    revElasticSearch7 = '7.6.2'
     revEmbeddedRedis = '0.6'
     revEurekaClient = '1.10.10'
     revGroovy = '2.5.13'

es7-persistence/README.md

@@ -0,0 +1,86 @@
+# ES7 Persistence
+
+This module provides ES7 persistence when indexing workflows and tasks.
+
+## ES Breaking changes
+
+From ES6 to ES7 there were significant breaking changes which affected ES7-persistence module implementation.
+* Mapping type deprecation
+* Templates API
+* TransportClient deprecation
+
+More information can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html
+
+
+## Build
+
+1. In order to use the ES7, you must change the following files from ES6 to ES7:
+
+https://github.com/Netflix/conductor/blob/main/build.gradle
+https://github.com/Netflix/conductor/blob/main/server/src/main/resources/application.properties
+
+In file:
+
+- /build.gradle
+
+change ext['elasticsearch.version'] from revElasticSearch6 to revElasticSearch7
+
+
+In file:
+
+- /server/src/main/resources/application.properties
+
+change conductor.elasticsearch.version from 6 to 7
+
+Also you need to recreate dependencies.lock files with ES7 dependencies. To do that delete all dependencies.lock files and then run: 
+
+```
+./gradlew generateLock updateLock saveLock
+```
+
+
+2. To use the ES7 for all modules include test-harness, you must change also the following files:
+
+https://github.com/Netflix/conductor/blob/main/test-harness/build.gradle
+https://github.com/Netflix/conductor/blob/main/test-harness/src/test/java/com/netflix/conductor/test/integration/AbstractEndToEndTest.java
+
+In file:
+
+- /test-harness/build.gradle
+
+* change module inclusion from 'es6-persistence' to 'es7-persistence'
+
+In file:
+
+- /test-harness/src/test/java/com/netflix/conductor/test/integration/AbstractEndToEndTest.java
+
+* change conductor.elasticsearch.version from 6 to 7
+* change DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch-oss").withTag("6.8.12") to DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch-oss").withTag("7.6.2")
+
+
+## Usage
+
+This module uses the following configuration options:
+
+* `conductor.elasticsearch.url` - A comma separated list of schema/host/port of the ES nodes to communicate with.
+Schema can be `http` or `https`. If schema is ignored then `http` transport will be used;
+Since ES deprecated TransportClient, conductor will use only the  REST transport protocol.
+* `conductor.elasticsearch.indexPrefix` - The name of the workflow and task index.
+Defaults to `conductor`
+* `conductor.elasticsearch.asyncWorkerQueueSize` - Worker Queue size used in executor service for async methods in IndexDao 
+Defaults to `100`
+* `conductor.elasticsearch.asyncMaxPoolSize` - Maximum thread pool size in executor service for async methods in IndexDao        
+Defaults to `12`
+* `conductor.elasticsearch.asyncBufferFlushTimeout` - Timeout (in seconds) for the in-memory to be flushed if not explicitly indexed
+Defaults to `10`
+
+### BASIC Authentication
+If you need to pass user/password to connect to ES, add the following properties to your config file
+* conductor.elasticsearch.username
+* conductor.elasticsearch.password
+
+Example
+```
+conductor.elasticsearch.username=someusername
+conductor.elasticsearch.password=somepassword
+```

es7-persistence/build.gradle

@@ -0,0 +1,62 @@
+/*
+ *  Copyright 2021 Netflix, Inc.
+ *  <p>
+ *  Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ *  the License. You may obtain a copy of the License at
+ *  <p>
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *  <p>
+ *  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ *  an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ *  specific language governing permissions and limitations under the License.
+ */
+
+plugins {
+    id 'com.github.johnrengelman.shadow' version '6.1.0'
+    id 'java'
+}
+
+configurations {
+    // Prevent shaded dependencies from being published, while keeping them available to tests
+    shadow.extendsFrom compileOnly
+    testRuntime.extendsFrom compileOnly
+}
+
+ext['elasticsearch.version'] = revElasticSearch7
+
+dependencies {
+    implementation project(':conductor-common')
+    implementation project(':conductor-core')
+
+    compileOnly 'org.springframework.boot:spring-boot-starter'
+
+    implementation "commons-io:commons-io:${revCommonsIo}"
+    implementation "org.apache.commons:commons-lang3"
+    // SBMTODO: remove guava dep
+    implementation "com.google.guava:guava:${revGuava}"
+
+    implementation "com.fasterxml.jackson.core:jackson-databind"
+    implementation "com.fasterxml.jackson.core:jackson-core"
+
+    implementation "org.elasticsearch.client:elasticsearch-rest-client"
+    implementation "org.elasticsearch.client:elasticsearch-rest-high-level-client"
+
+    testImplementation "org.awaitility:awaitility:${revAwaitility}"
+    testImplementation "org.testcontainers:elasticsearch:${revTestContainer}"
+    testImplementation project(':conductor-common').sourceSets.test.output
+}
+
+// Drop the classifier and delete jar task actions to replace the regular jar artifact with the shadow artifact
+shadowJar {
+    configurations = [project.configurations.shadow]
+    classifier = null
+
+    // Service files are not included by default.
+    mergeServiceFiles {
+        include 'META-INF/services/*'
+        include 'META-INF/maven/*'
+    }
+}
+
+jar.enabled = false
+jar.dependsOn shadowJar