Pipeline Configuration Transformation #4446

srikanthjg · 2024-04-19T17:27:52Z

Description

Support pipeline configuration transformation based on rules and templates.

Rules for a transformation will be defined in a yaml file. It will be checked for every pipeline. If rules apply, a corresponding template is chosen. The template defines the transformation for that pipeline. Currently if there are multiple sub-pipelines, only one pipeline is chosen for which the rules apply.

Template definitions uses JsonPath to index nodes in the user provided configuration.

Current Assumptions:

In the template definition, Deep scan or recursive expressions like$.. is NOT supported. Always use a more specific expression. In the event specific variables in a path are not known, use wildcards.
User could provide multiple pipelines in their user config but there can be only one pipeline that can support transformation.
There cannot be multiple transformations in a single pipeline.
<< .. >> is the placeholder in the template.
<<pipeline-name>> is handled differently as compared to other placeholders
as other placeholders are jsonPaths, this just replaces placeholder with the pipeline Name that needs transformation.

Issues Resolved

Resolves #4444

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

srikanthjg · 2024-04-19T21:18:04Z

data-prepper-pipeline-parser/src/resources/templates/documentdb-template.yaml

@@ -0,0 +1,69 @@
+"{{pipeline-name}}-writer":


This template is not the final version, needs to be finalized. Check the test/resources/templates/testSource for the templates i used to test

acknowledgments: false -> This should be true.

srikanthjg · 2024-04-19T23:59:30Z

...line-parser/src/main/java/org/opensearch/dataprepper/pipeline/parser/rule/RuleEvaluator.java

+     * @return RuleEvaluatorResult
+     */
+    private RuleEvaluatorResult isDocDBSource(PipelinesDataFlowModel pipelinesModel) {
+        PLUGIN_NAME = "documentdb";


This is only a temporary implementation, will move towards scanning folder and fetching and applying rule file dynamically.

dlvenable

I made some high-level comments. I'll also try to dive into some lower-level details later.

data-prepper-api/build.gradle

data-prepper-core/src/main/java/org/opensearch/dataprepper/AbstractContextManager.java

data-prepper-core/src/main/java/org/opensearch/dataprepper/parser/PipelineTransformer.java

...r/src/main/java/org/opensearch/dataprepper/pipeline/parser/PipelinesDataflowModelParser.java

data-prepper-pipeline-parser/src/main/java/org/opensearch/dataprepper/pipeline/parser/README.md

...in/java/org/opensearch/dataprepper/pipeline/parser/transformer/DynamicConfigTransformer.java

...org/opensearch/dataprepper/pipeline/parser/transformer/PipelineConfigurationTransformer.java

...core/src/main/java/org/opensearch/dataprepper/parser/config/PipelineParserConfiguration.java

...arser/src/main/java/org/opensearch/dataprepper/pipeline/parser/rule/RuleEvaluatorResult.java

kkondaka · 2024-04-22T23:43:04Z

...rser/src/main/java/org/opensearch/dataprepper/pipeline/parser/rule/RuleTransformerModel.java

+import java.util.List;
+
+@Data
+public class RuleTransformerModel {


Should this be RuleTransformerModelConfig?

can address this nitpick in the subsequent pr.

kkondaka · 2024-04-23T00:06:11Z

...in/java/org/opensearch/dataprepper/pipeline/parser/transformer/DynamicConfigTransformer.java

+     * @param json
+     * @return Map<String, List<String>> , K:placeholder, V: list of jsonPath in templateJson
+     * @throws IOException
+     */


Probably good idea to document example input args and expected output which helps in understanding the code better. (This comment is for all functions in this file)

can address this nitpick in the subsequent pr.

left few comments to be addressed

dlvenable

@srikanthjg , Your DCO is failing. You have some commits from a merge in here. Please use a rebase from main instead.

Other than that, this should be good to merge. I think the rest of the changes will now be internal to the transformation code.

dlvenable · 2024-04-24T21:38:10Z

@srikanthjg , Please also add a DCO to your commits.

Please see:

https://github.com/opensearch-project/data-prepper/blob/main/CONTRIBUTING.md#developer-certificate-of-origin-dco

Signed-off-by: srigovs <srigovs@amazon.com>

…cDB with big template Signed-off-by: srigovs <srigovs@amazon.com>

Signed-off-by: srigovs <srigovs@amazon.com>

templates and rules Signed-off-by: srigovs <srigovs@amazon.com>

Signed-off-by: srigovs <srigovs@amazon.com>

dlvenable

Looks good. Thanks for this significant contribution!

srikanthjg requested review from chenqi0805, engechas, graytaylor0, dinujoh, kkondaka, asifsmohammed, KarstenSchnitter, dlvenable and oeyh as code owners April 19, 2024 17:27

dlvenable mentioned this pull request Apr 19, 2024

Bootstrap the RuleEngine package #4442

Closed

1 task

srikanthjg commented Apr 19, 2024

View reviewed changes

dlvenable requested changes Apr 22, 2024

View reviewed changes

kkondaka previously approved these changes Apr 23, 2024

View reviewed changes

dlvenable requested changes Apr 24, 2024

View reviewed changes

srikanthjg force-pushed the transformation branch from d1f6fd7 to f65594b Compare April 24, 2024 22:06

srikanthjg added 12 commits April 24, 2024 17:02

Adding templates

f43aaf8

Signed-off-by: srigovs <srigovs@amazon.com>

Added Dynamic yaml transformer

7f80316

Signed-off-by: srigovs <srigovs@amazon.com>

Added Rule evaluator

b549e41

Signed-off-by: srigovs <srigovs@amazon.com>

Added rule evaluator

769a3ae

Signed-off-by: srigovs <srigovs@amazon.com>

Added json walk

1817b56

Signed-off-by: srigovs <srigovs@amazon.com>

Add transformation logic

1ba89fb

Signed-off-by: srigovs <srigovs@amazon.com>

Add dynamic rule

7ac2c0a

Signed-off-by: srigovs <srigovs@amazon.com>

Almost working

2060882

Signed-off-by: srigovs <srigovs@amazon.com>

Adding multiple pipelines part1

fc50983

Signed-off-by: srigovs <srigovs@amazon.com>

Adding multiple pipelines part2-incomplete

a63961d

Signed-off-by: srigovs <srigovs@amazon.com>

Works e2e for 1 pipeline

8b877b5

Signed-off-by: srigovs <srigovs@amazon.com>

Added multi pipeline template and pipelinemodel support, works for do…

97472c0

…cDB with big template Signed-off-by: srigovs <srigovs@amazon.com>

srikanthjg added 10 commits April 24, 2024 17:04

added tests for models and fixed beans, one more fix needed for bean

40750e3

Signed-off-by: srigovs <srigovs@amazon.com>

Fixed IT and beans

abe79a1

Signed-off-by: srigovs <srigovs@amazon.com>

Update bean to have only pipelineDataModel and not parser

422e3ab

Signed-off-by: srigovs <srigovs@amazon.com>

Add banner

b89e3a9

Signed-off-by: srigovs <srigovs@amazon.com>

Code cleanup and add comments

147cd86

Signed-off-by: srigovs <srigovs@amazon.com>

Support user pipeline configuration dynamic transformation based on

6d4ed2b

templates and rules Signed-off-by: srigovs <srigovs@amazon.com>

Address comments

a7badaf

Signed-off-by: srigovs <srigovs@amazon.com>

Added Function Call support in templates

8eb6894

Signed-off-by: srigovs <srigovs@amazon.com>

Added Function Call support in templates

b0544f2

Signed-off-by: srigovs <srigovs@amazon.com>

Modify documentDB template.

5b4825f

Signed-off-by: srigovs <srigovs@amazon.com>

srikanthjg force-pushed the transformation branch from 24d4c44 to 5b4825f Compare April 25, 2024 00:06

srikanthjg added 2 commits April 24, 2024 17:31

Code clean up

d04768c

Signed-off-by: srigovs <srigovs@amazon.com>

Code clean up

36b1dba

Signed-off-by: srigovs <srigovs@amazon.com>

dlvenable approved these changes Apr 25, 2024

View reviewed changes

kkondaka approved these changes Apr 25, 2024

View reviewed changes

kkondaka merged commit e590fde into opensearch-project:main Apr 25, 2024
47 checks passed

kkondaka added this to the v2.8 milestone May 14, 2024

srikanthjg mentioned this pull request Jun 5, 2024

Dynamic Rule Detection #4600

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipeline Configuration Transformation #4446

Pipeline Configuration Transformation #4446

srikanthjg commented Apr 19, 2024 •

edited

Loading

srikanthjg Apr 19, 2024

sharraj Apr 20, 2024

srikanthjg Apr 19, 2024

dlvenable left a comment

kkondaka Apr 22, 2024

srikanthjg Apr 24, 2024

kkondaka Apr 23, 2024

srikanthjg Apr 24, 2024

dlvenable left a comment

dlvenable commented Apr 24, 2024

dlvenable left a comment

Pipeline Configuration Transformation #4446

Pipeline Configuration Transformation #4446

Conversation

srikanthjg commented Apr 19, 2024 • edited Loading

Description

Issues Resolved

Check List

srikanthjg Apr 19, 2024

Choose a reason for hiding this comment

sharraj Apr 20, 2024

Choose a reason for hiding this comment

srikanthjg Apr 19, 2024

Choose a reason for hiding this comment

dlvenable left a comment

Choose a reason for hiding this comment

kkondaka Apr 22, 2024

Choose a reason for hiding this comment

srikanthjg Apr 24, 2024

Choose a reason for hiding this comment

kkondaka Apr 23, 2024

Choose a reason for hiding this comment

srikanthjg Apr 24, 2024

Choose a reason for hiding this comment

dlvenable left a comment

Choose a reason for hiding this comment

dlvenable commented Apr 24, 2024

dlvenable left a comment

Choose a reason for hiding this comment

srikanthjg commented Apr 19, 2024 •

edited

Loading