Data stream datasets are not picked up dynamically for Input type packages

[ShourieG]

Currently for Input packages, elastic-package does not seem to be picking up datastream.dataset defined under vars:
Example -

service: google-cloud-storage-emulator
input: gcs
vars:
  project_id: testproject
  alternative_host: "http://{{Hostname}}:{{Port}}"
  data_stream.dataset: google_cloud_storage.generic
  buckets: |
    - name: testbucket
      poll: true
      poll_interval: 15s

elastic-package always defaults to the convention of {{package_dir_name}}.{{policy_template_name}} for input packages, since input packages lack the data stream directory. This causes the issue where testing input packages with custom data streams that break the {{package_dir_name}}.{{policy_template_name}} convention, to become impossible.

[jsoriano]

convention of {{package_dir_name}}.{{policy_template_name}} for input packages

@ShourieG does it work when the convention is followed? Is there a use case where we need to define tests with different datasets?

[jsoriano]

(asking to prioritize in case this completely blocks testing of input packages or if it is a papercut)

[ShourieG]

@jsoriano It works when convention is followed since current the code defaults to this convention always when no data stream folders exist. The bug is because the config values for datastream.dataset is not picked up at all in case of an input package. There are no immediate scenarios which mandate the use of custom datasets but the expection is that elastic-package should atleast honor the config values for datastream.dataset similar to normal integration packages.

[jsoriano]

elastic-package should atleast honor the config values for datastream.dataset similar to normal integration packages.

From what I see in the code (here), elastic-package neither honors the config value for datastream.dataset for integration packages, it uses the names of the package and the data stream, or the value overriden in the data stream manifest. But it is true that for input packages setting this is mandatory for users.

Also looking to the code, it looks like for input packages, data_stream.dataset is set to the value expected by convention if no other value is set (here). I guess this covers most of the cases.

So what would be missing would be to support the cases in which the developer wants or needs to override the dataset value. But I guess this is not so common?

[ShourieG]

@jsoriano So the difference here is for integration package the datastream overridden in the manifest is still honoured but for input packages, it does not do so , irrespective of what you have set in the manifest, it will always default to the convention.

[jsoriano]

In principle it is not possible to define a dataset in input packages manifests. It can only be configured by the user by configuration. elastic-package uses {{package_name}}.{{policy_template_name}} as default by convention.

[ShourieG]

@jsoriano, I see, but just wanted to know why this restriction is present since when we are testing we would want as much of a real world scenario as possible. Having the tests honour custom datasets would end up enriching the tests in the long run I feel would it not ?

[jsoriano]

Yes, agree, only that I see this more as an enhancement than an actual issue.

[jsoriano]

We need to increase the priority of this, after elastic/kibana#182925 this integration cannot be tested because it would need to remove the -s in the dataset.

I will try to give a quick look to this today.

cc @kpollich