Pipeline test fails when target datasets in routing rules contain an expression

[zmoog]

elastic-package version: 0.85.0

Given the following routing_rules.yml file:

- source_dataset: kubernetes.container_logs
  rules:
    - target_dataset:
        - "{{kubernetes.labels.elastic_co/dataset}}"
        - "{{data_stream.dataset}}"
        - kubernetes.container_logs
      namespace:
        - "{{kubernetes.labels.elastic_co/namespace}}"
        - "{{data_stream.namespace}}"
        - default
      if: "ctx?.kubernetes?.labels != null"

and test-common-config.yml file:

fields:
  data_stream:
    dataset: kubernetes.container_logs
    namespace: default
  kubernetes:
    labels:
      elastic_co/dataset: kubernetes.container_logs.nginx
      elastic_co/namespace: nginx
      app_kubernetes_io/version: "v0.1.0"
      app_kubernetes_io/name: "myservice"

and test-nginx.log file:

2023/07/25 15:24:11 [notice] 1#1: start worker process 33
2023/07/25 15:24:11 [notice] 1#1: start worker process 34
2023/07/25 15:24:11 [notice] 1#1: start worker process 35
2023/07/25 15:24:11 [notice] 1#1: using the "epoll" event method

When I run elastic-package test pipeline -v -g command I get the following error:

2023/08/10 19:23:46 DEBUG Enable verbose logging
Run pipeline tests for the package
--- Test results for package: kubernetes - START ---
FAILURE DETAILS:
kubernetes/container_logs test-nginx.log:
[0] field "data_stream.dataset" should have value in ["{{kubernetes.labels.elastic_co/dataset}}" "{{data_stream.dataset}}" "kubernetes.container_logs"], it has "kubernetes.container_logs.nginx"


╭────────────┬────────────────┬───────────┬────────────────┬─────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE    │ DATA STREAM    │ TEST TYPE │ TEST NAME      │ RESULT                                                                      │ TIME ELAPSED │
├────────────┼────────────────┼───────────┼────────────────┼─────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ kubernetes │ container_logs │ pipeline  │ test-nginx.log │ FAIL: test case failed: one or more problems with fields found in documents │      2.617ms │
╰────────────┴────────────────┴───────────┴────────────────┴─────────────────────────────────────────────────────────────────────────────┴──────────────╯
--- Test results for package: kubernetes - END   ---
Done
Error: one or more test cases failed

It seems elastic-package is interpreting the target_dataset values as literals and not expressions, so it's not evaluating the "{{kubernetes.labels.elastic_co/dataset}}" expression in the into its actual value kubernetes.container_logs.nginx.

[kaiyan-sheng]

Thanks for finding this issue! Two problems here:

1. we need to validate the field value instead of the field name when using {{datastream.dataset}}

[0] field "data_stream.dataset" should have value in ["{{data_stream.dataset}}"], it has "test_dataset"

2. when writing into -generated.json file, we have duplicate fields. for example:

            "data_stream": {
                "dataset": "test_dataset",
                "namespace": "test_namespace"
            },
            "data_stream.dataset": "test_dataset",
            "data_stream.namespace": "test_namespace",

[zmoog]

@jsoriano, I'm more than happy to work on this (it's a dependency for rerouting k8s containers logs based on pod labels).

Any advice about how to make this happen? I mean similar stuff, existing libraries in elastic-package — or anything that can help someone new to this codebase? 🙇

[jsoriano]

I'm more than happy to work on this (it's a dependency for rerouting k8s containers logs based on pod labels).

@zmoog thanks! Let me assign the issue to you.

Any advice about how to make this happen? I mean similar stuff, existing libraries in elastic-package — or anything that can help someone new to this codebase? 🙇

Sadly we don't have documentation about the elastic-package codebase, but you could look to the PR that added this feature, here: #1372
In any case it looks like you go in the good direction in your draft PR 🙂