[meta] Pipeline Java Execution important issues

[colinsurprenant]

This is a meta issue about some important Pipeline Java Execution issues.

Priority Issues

• Java execution may start inputs before compilation of filters and outputs has completed Java execution may start inputs before compilation of filters and outputs has completed. #11170 - Fixed in start inputs only when all WorkerLoop are fully initialized #11492
• Java execution reorders events even with 1 pipeline worker Java execution reorders events even with 1 pipeline worker #10938
  • Abandoned tentative fix Preserve input event ordering with single pipeline worker #11020
  • Incomplete POC [POC][DISCUSS] preserve batched events order #11099 (does not solve the [complex aggregate filter scenario]
    (https://gist.github.com/yaauie/82e199687c490a2e04fe1f0c9d9a7fee))
  • New solution PR Add support for pipeline.ordered setting for java execution #11524
• Java execution startup is potentially slower for some configurations
• fix for regression introduced by Move class caching from ComputeStepSyntaxElement to CompiledPipeline #11482 with revert #11482 and fix redundant code generation #11564

Followup Issues

• improve pipeline.ordered setting to rebatch before outputs improve pipeline.ordered setting to rebatch before outputs #11550 - Fixed in Rebatch after filters when using pipeline.ordered #11710
• Document event ordering guaranties or lack thereof with multiple worker versus single worker - [Doc]Add section for conceptual info #11715
• Add integration harness around the expected compilation time which should be similar between single or multiple workers. Add integration test on JEE expected compilation time  #12476

Other issues

• quoted plugin options crashes logstash When using Java Execution Engine, using quoted plugin directives crashes Logstash #11222 - Fixed in support quoted plugin option keys #11688

[colinsurprenant]

I have looked into the "Java execution startup is potentially slower for some configurations #11105" issue and as I commented in that issue, it is not potentially slower, the slowdown is in fact a factor of the number of workers, see #11105 (comment).

I will continue investigating this.

[colinsurprenant]

I have created 2 PRs with alternate implementations to solve the class compilation caching problem leading to recompiling everything per worker; the first #11479 uses the existing global cache but fixes the cache key to correctly reuse compiled classes. There are some potential problems with pipeline reloading and cache invalidation with this implementation and another PR #11482 uses a different strategy by moving the class caching at the pipeline level which solves both the potential invalidation and pipeline reloading problems.

[colinsurprenant]

Good news on the inputs starting before the workers are initialized: it is easier than I thought to fix, now just need to see if we make that configurable. Will push PR soon.

[colinsurprenant]

#11482 was merged and should considerably improve java execution pipeline compilation time as it will not be multiplied by the number of workers anymore. Fix will be included in 7.6.0 and 7.5.2.

[colinsurprenant]

PR to fix worker initialization sequence in #11492. Should we make that behaviour optional?

[colinsurprenant]

#11492 is merged and will be included in 7.6.0 and 7.5.2. We decided that it was indeed a bug fix and that adding an option to eagerly start inputs before workers are fully initialized was a feature to evaluate in #11493