Support routing_rules.yml and ingest pipeline test for reroute processor

[kaiyan-sheng]

What does this PR do?

This PR does two things:

1. Support routing_rules.yml file when running pipeline test using command elastic-package test pipeline -d <datastream> --generate -v.

routing_rules.yml now can be read and take into account when running the test pipeline command. The file is read as yml and convert into reroute processors. Then these reroute processors are attached under the processors section from the ingest pipeline.

For example:

- source_dataset: awsfirehose.log
  rules:
    - target_dataset: aws.cloudtrail
      if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
      namespace:
        - "{{labels.data_stream.namespace}}"
        - default

routing_rules.yml looks like the example above will be translated into:

- reroute:
      tag: awsfirehose.log
      if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
      dataset: aws.cloudtrail
      namespace: ["{{labels.data_stream.namespace}}", default]

2. Support ingest pipeline test for reroute processors when they are added under elasticsearch/ingest_pipeline directly.

• This PR added the index name for testing based on the dataset so we don't see error like this:

[0] unexpected pipeline error: invalid data stream name: [_index]; must follow naming scheme <type>-<dataset>-<namespace>

• When checking the value for data_stream.dataset, it's read from the reroute processors for the new target dataset. Because reroute processor will change the value of this field.

How to test this PR locally

I've been testing this PR locally with elastic/integrations#7146. After make build elastic package locally, I ran elastic-package test pipeline -d log --generate -v and under integrations/packages/awsfirehose/data_stream/log/_dev/test/pipeline/test-cloudtrail.json-expected.json, you can see the data_stream.dataset value changed to aws.cloudtrail and data_stream.namespace changed to default. These are the two fields I defined under reroute processor:

  - reroute:
      if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
      dataset: aws.cloudtrail
      namespace: default

Same thing works for defining it under routing_rules.yml

- source_dataset: awsfirehose.log
  rules:
    - target_dataset: aws.cloudtrail
      if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
      namespace:
        - "{{labels.data_stream.namespace}}"
        - default

[jsoriano]

This PR is to support ingest pipeline test for reroute processor.

Hey @kaiyan-sheng,

This PR adds support to test the reroute processors defined on ingest pipelines, right? But the recommended way to add routing rules is to use the routing_rules.yml file (as defined in elastic/package-spec#514).

I think that appart of these changes, for pipeline tests, we would need to make elastic-package to install the routing rules in a way that they apply when using the simulate API.

It would be also great to add a test package here.

[jsoriano]

Also, by reading the routing_rules.yml file, tests might know on what datasets data stream to look for the data.

[felixbarny]

This PR adds support to test the reroute processors defined on ingest pipelines, right? But the recommended way to add routing rules is to use the routing_rules.yml file (as defined in elastic/package-spec#514).

The routing rules in routing_rules.yml will be translated into reroute processors that are added in the package's pipeline. So in theory, they should be tested by test pipeline command. If not, that's an issue we should fix.

[jsoriano]

This PR adds support to test the reroute processors defined on ingest pipelines, right? But the recommended way to add routing rules is to use the routing_rules.yml file (as defined in elastic/package-spec#514).

The routing rules in routing_rules.yml will be translated into reroute processors that are added in the package's pipeline. So in theory, they should be tested by test pipeline command. If not, that's an issue we should fix.

Yes, they are translated to reroute processors when installed through Fleet, but I think that currently elastic-package test pipelines only installs the pipelines, it doesn't install the package, so it is probably not installing the routing rules. We should probably change this.

[kaiyan-sheng]

@jsoriano Yep it only looks for the yml files under elasticsearch/ingest_pipeline for pipelines right now. Can I keep this PR for only supporting ingest pipelines for reroute processors and then have a separate PR for supporting routing_rules.yml?

[zmoog]

More than happy to help with adding support for the routing-rules.yml to the pipeline testing; I also need this for the Kubernetes integration. I'm available for both implementation and review.

[jsoriano]

Thanks for adding tests! It looks like expectation files need to be regenerated.

[jsoriano]

Thanks!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: d3444e8

History

• 💚 Build #1268 succeeded d5f3077
• 💚 Build #1267 succeeded 17000b9
• 💔 Build #1250 failed 19feb7a
• 💔 Build #1249 failed eacd48e
• 💔 Build #1248 failed bc82c16
• 💔 Build #1246 failed 2a2756f

cc @kaiyan-sheng