Pipeline Configuration Transformation

[srikanthjg]

Is your feature request related to a problem? Please describe.
Current DataPrepper configurations are very verbose and very tedious to write manually. With increasing number of processors and functionality in the DataPrepper, it would be difficult for the customers to know the correct processors to use with appropriate config options in correct sequence.
Secondly, some customer use cases might require multiple data-prepper pipelines even though their high level requirement is not very complex.

Describe the solution you'd like
Support transformation of pipeline configuration from user provided configuration to
a transformed configuration based on template and rules.

User config

simple-pipeline:
  source:
    someSource:
      hostname: "database.example.com"
      port: "27017"
  sink:   
    - opensearch:
       hosts: ["https://search-service.example.com"]
       index: "my_index"

Template

"<<pipeline-name>>-transformed":
  source: "<<$.*.someSource>>"
  sink:
    - opensearch:
       hosts: "<<$.*.sink[?(@.opensearch)].opensearch.hosts>>"
       port: "<<$.*.someSource.documentdb.port>>"
       index: "<<$.*.sink[0].opensearch.index>>"
       aws:
        sts_role_arn: "arn123"
        region: "us-test-1"
       dlq:
          s3:
              bucket: "test-bucket"

Rule

apply_when:
  - "$..source.someSource"

Expected Transformed Config

simple-pipeline-transformed:
  source:
    someSource:
      hostname: "database.example.com"
      port: "27017"
  sink:
    - opensearch:
        hosts: ["https://search-service.example.com"]
        port: "27017"
        index: "my_index"
        aws:
          sts_role_arn: "arn123"
          region: "us-test-1"
        dlq:
          s3:
            bucket: "test-bucket"