Skip to content

Honor data_stream.dataset in input packages #1851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 22, 2024
Merged

Honor data_stream.dataset in input packages #1851

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions internal/testrunner/runners/system/runner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -988,17 +988,25 @@ func (r *runner) prepareScenario(ctx context.Context, config *testConfig, svcInf

// Delete old data
logger.Debug("deleting old data in data stream...")

// Input packages can set `data_stream.dataset` by convention to customize the dataset.
dataStreamDataset := ds.Inputs[0].Streams[0].DataStream.Dataset
if scenario.pkgManifest.Type == "input" {
v, _ := config.Vars.GetValue("data_stream.dataset")
if dataset, ok := v.(string); ok && dataset != "" {
dataStreamDataset = dataset
}
}
scenario.dataStream = fmt.Sprintf(
"%s-%s-%s",
ds.Inputs[0].Streams[0].DataStream.Type,
ds.Inputs[0].Streams[0].DataStream.Dataset,
dataStreamDataset,
ds.Namespace,
)

componentTemplatePackage := fmt.Sprintf(
"%s-%s@package",
ds.Inputs[0].Streams[0].DataStream.Type,
ds.Inputs[0].Streams[0].DataStream.Dataset,
dataStreamDataset,
)

r.wipeDataStreamHandler = func(ctx context.Context) error {
Expand Down Expand Up @@ -1431,6 +1439,12 @@ func (r *runner) validateTestScenario(ctx context.Context, result *testrunner.Re
}
expectedDatasets = []string{expectedDataset}
}
if scenario.pkgManifest.Type == "input" {
v, _ := config.Vars.GetValue("data_stream.dataset")
if dataset, ok := v.(string); ok && dataset != "" {
expectedDatasets = append(expectedDatasets, dataset)
}
}

fieldsValidator, err := fields.CreateValidatorForDirectory(r.dataStreamPath,
fields.WithSpecVersion(scenario.pkgManifest.SpecVersion),
Expand Down Expand Up @@ -1652,8 +1666,14 @@ func createInputPackageDatastream(
// Add policyTemplate-level vars.
vars := setKibanaVariables(policyTemplate.Vars, config.Vars)
if _, found := vars["data_stream.dataset"]; !found {
dataStreamDataset := dataset
v, _ := config.Vars.GetValue("data_stream.dataset")
if dataset, ok := v.(string); ok && dataset != "" {
dataStreamDataset = dataset
}

var value packages.VarValue
value.Unpack(dataset)
value.Unpack(dataStreamDataset)
vars["data_stream.dataset"] = kibana.Var{
Value: value,
Type: "text",
Expand Down
1 change: 1 addition & 0 deletions test/packages/parallel/sql_input/_dev/test/system/test-default-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ vars:
hosts:
- root:test@tcp({{Hostname}}:{{Port}})/
sql_query: "SHOW GLOBAL STATUS LIKE 'Innodb_data%';"
data_stream.dataset: sql_input.test
2 changes: 2 additions & 0 deletions test/packages/parallel/sql_input/agent/input/input.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
metricsets: ["query"]
data_stream:
dataset: {{data_stream.dataset}}
Comment on lines +2 to +3
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be required in input packages to actually leverage this option. I think we should handle this in a different way. Ideally Fleet should add this if this is required in all input packages. cc @kpollich @mrodm

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think you are right. Fleet is enforcing this as a requirement, so Fleet should add this to the input package if it isn't explicitly set as a "default". Dynamically altering the handlebars template before rendering it isn't something we've done before in Fleet, but I don't think it will be more than some string manipulation. There's potential for complexity, though, to be sure. I created elastic/kibana#184008 to capture this on the Fleet side.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍

I think this config could be appended if not already set.

period: {{period}}
hosts:
{{#each hosts}}
Expand Down