Wt 23007 support es7 in nfx v3

[vukor]

Pull Request type

• Bugfix
• Feature
• Refactoring (no functional changes, no api changes)
• Build related changes
• Other (please describe):

Changes in this PR

This pr is copied from Netflix#2324

[github-actions]

Unit Test Results

116 files  ±0  116 suites  ±0   12m 26s ⏱️ ±0s
921 tests ±0  909 ✔️ ±0  12 💤 ±0  0 ❌ ±0 

Results for commit 39a860b. ± Comparison against base commit 39a860b.

♻️ This comment has been updated with latest results.

[vukor]

I've deployed separate apps in tst cluster + AWS ES7:

• https://nfx-conductor-v3-es7.tst.ai.locusdev.net/
• https://nfx-conductor-server-v3-es7.tst.ai.locusdev.net/

@marcocrasso could you please check elasticsearch integration? Currently there are no indices in ES (I've sent you ES creds to wickr)

[marcocrasso]

@vukor
I've been reviewing this PR. I pull the branch and run in dev mode, by adding into server/src/main/resources/application.properties the next three options:

conductor.elasticsearch.url=https://vpc-ai-nfx-conductor-tst-gvdjf6rgtk6w5llenpm4kf4sqi.us-east-1.es.amazonaws.com
conductor.elasticsearch.username=elastic
conductor.elasticsearch.password=<<< removed >>>

and hit ./gradlew server

Then I pull ai-recsys-core-index@main modified the health check and hit:
export CONDUCTOR_URL=http://localhost:8080/api/
make run

The above registers one task definition and one workflow definition too. Then in another shell I run make start_wf and the workflow run successfully but in the Conductor Server log, I saw an exception which tells that logs of the tasks couldn't be indexed. When I checked the ES7 indeces I confirmed that one was missing.

✗ curl -u 'elastic:<<>>' https://vpc-ai-nfx-conductor-tst-gvdjf6rgtk6w5llenpm4kf4sqi.us-east-1.es.amazonaws.com/_cat/indices
green open conductor_task jZzcYMQOQ8KjmTtfeoZNZA 5 1 1 1 59.1kb 29.5kb
green open conductor_workflow N4BankGlTiu5SHLwZbLinA 5 1 1 2 78.4kb 39.2kb

The exception is below, have you been looking at index creation? I can take a look tomorrow.

185826 [http-nio-8080-exec-8] ERROR com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7 [] - Failed to index task execution logs for tasks: [98e7c11a-cfc6-440d-85cf-a40bfffe6172, 98e7c11a-cfc6-440d-85cf-a40bfffe6172, 98e7c11a-cfc6-440d-85cf-a40bfffe6172] java.lang.RuntimeException: Operation 'addTaskExecutionLogs:Indexing task execution logs' failed after retrying 3 times, retry limit 3 at com.netflix.conductor.common.utils.RetryUtil.retryOnException(RetryUtil.java:126) ~[conductor-common-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7.addTaskExecutionLogs(ElasticSearchRestDAOV7.java:543) [conductor-es7-persistence-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.netflix.conductor.core.orchestration.ExecutionDAOFacade.addTaskExecLog(ExecutionDAOFacade.java:485) [conductor-core-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.netflix.conductor.core.execution.WorkflowExecutor.updateTask(WorkflowExecutor.java:1030) [conductor-core-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.netflix.conductor.service.ExecutionService.updateTask(ExecutionService.java:216) [conductor-core-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.netflix.conductor.service.TaskServiceImpl.updateTask(TaskServiceImpl.java:125) [conductor-core-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.netflix.conductor.service.TaskServiceImpl$$FastClassBySpringCGLIB$$d78c3007.invoke(<generated>) [conductor-core-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) [spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) [spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:119) [spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) [spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) [spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE] at com.netflix.conductor.service.TaskServiceImpl$$EnhancerBySpringCGLIB$$2928e612.updateTask(<generated>) [conductor-core-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.netflix.conductor.rest.controllers.TaskResource.updateTask(TaskResource.java:78) [conductor-rest-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.2.7.RELEASE.jar:5.2.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) [spring-boot-actuator-2.3.1.RELEASE.jar:2.3.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.36.jar:9.0.36] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.36.jar:9.0.36] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.36.jar:9.0.36] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: index is missing;2: index is missing;3: index is missing; at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:447) ~[elasticsearch-7.10.1.jar:7.10.1] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1] at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:537) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1] at com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7.lambda$addTaskExecutionLogs$5(ElasticSearchRestDAOV7.java:545) ~[conductor-es7-persistence-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-2.0.0.jar:?] at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[guava-retrying-2.0.0.jar:?] at com.netflix.conductor.common.utils.RetryUtil.retryOnException(RetryUtil.java:115) ~[conductor-common-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] ... 70 more 185949 [pool-21-thread-1] INFO com.netflix.dyno.queues.redis.RedisDynoQueue [] - processUnacks() will NOT be atomic.

[marcocrasso]

Follow-up, I left a comment to the original author Netflix#2324 (comment)

[marcocrasso]

I've committed one change in es7-persistence module, which creates all required indexes. It is working, we can merge. Thanks Anton, great work!